Segovia was a simple day trip from Madrid. I would highly recommend you to go on your own given how completely accessible it is. There are many organized trips out there that would combine Segovia, Avila and Toledo with different combinations and permutations, it’s very attractive for someone who has less time, but you do waste time on the bus (since it’s slower than high speed train) and you’ll see less. I did Segovia and Toledo on different days in order to have enough time.
Segovia definitely took less time than Toledo purely from distance perspective and how long it took to tour the major attractions, I’d say half a day might be sufficient if you don’t stop a lot.