The disciples knew that Jesus had the authority to guard his Father's house (AKA the temple), but not everyone is convinced. Some of the people asked Jesus for a sign to demonstrate that he could do these things.
Jesus responded that his "sign" was rebuilding the temple after destroying it (in three days). The people highly doubted that could even happen because temple-building takes much more than three days, but as John stated later, Jesus was referring to His body (AKA what he would later do on the cross). This disciples later got the reference, too.
Since Jesus can demonstrate that the "temple" refers to the body (not to mention that others like the disciples can see the figurative meaning), it's more probable that the other people misinterpreted Jesus's words. But even then, considering that Jesus can do miracles like water-to-wine and dove, it's also possible for Jesus to build temples in 3 days. It's likely that when the people didn't see Jesus as God, they couldn't see that he could build temples in 3 days or do temple cleanup or other miracles. Our view on who God is affects what He can do, so it's good to find out who God really is.