The 11-member delegation, however, was not allowed to proceed beyond the drop gate of army headquarters of the Uri brigade, 20-kms from the Line-of Control.
The delegation also contacted the sub-divisional police officer for the permission but he denied, saying it was the prerogative of the army.
The Kashmiri Pandits, under the banner of Pir Panjal organisation, chanting slogans 'Sharda Mata ki Jai' left from Durga Nag temple here for Kaman post in border town of Uri.
"Ours is a peace mission. We have been the worst sufferer of the animosity between India and Pakistan and our journey is an attempt to restore peace in the subcontinent," president of the organisation Ramesh Hangloo told PTI before embarking on the 120-km journey.
The KP leaders had requested India and Pakistan to allow them to visit the temple in PoK, but were denied permission.
The delegation was hopeful that they would be allowed to visit the temple in Atamuqam tehsil in Muzaffarabad.
Asked if they were not allowed to cross the LoC, they said its shows that the two governments were not serious about Confidence Building Measures (CBMs) announced so far.
Sharda temple, in Neelam (Kishanganga) valley just beyond the LoC in Pok, is located in a small village called Shardi near the confluence of Kishanganga and Madhumati rivers. It is one of most revered shrine of Kashmiri Pandits, and they want to go to undertake repair work of the temple.