You are viewing our Forum Archives. To view or take place in current topics click here.
How to check a large list of Indexed URL in search engine us
Posted:
How to check a large list of Indexed URL in search engine usPosted:
Status: Offline
Joined: Aug 28, 20159Year Member
Posts: 5
Reputation Power: 0
Status: Offline
Joined: Aug 28, 20159Year Member
Posts: 5
Reputation Power: 0
I am trying to check thousands of URLs from search engine index, If they are existing or not. I need a Java program which can trigger these url one at a time to the search engine and record the response If it exists or not. I already ask this question to some [ Register or Signin to view external links. ] forum. But still I am waiting for complete solution. Someone there who would be willing to spend some time to support me to solve this problem.
#2. Posted:
Status: Offline
Joined: Jun 29, 201311Year Member
Posts: 211
Reputation Power: 13
Status: Offline
Joined: Jun 29, 201311Year Member
Posts: 211
Reputation Power: 13
1) Create a class and method for checking the status of a website. Judge based on 200 OK response header on HttpUrlConnection.
2) Maybe declare the above as a task or as a runnable class.
3) Create like 4 threads that run in parallel to check the list of sites at the same time. This will decrease time taken to do the whole list. You can either split the list into how many number of threads you're gonna use or you can use an atomic/synchronized int that increments the index of the list each time an element is checked for validity (part 1).
2) Maybe declare the above as a task or as a runnable class.
3) Create like 4 threads that run in parallel to check the list of sites at the same time. This will decrease time taken to do the whole list. You can either split the list into how many number of threads you're gonna use or you can use an atomic/synchronized int that increments the index of the list each time an element is checked for validity (part 1).
- 0useful
- 0not useful
You are viewing our Forum Archives. To view or take place in current topics click here.