ProgrammingNeed help with url status check program
Posted:

ProgrammingNeed help with url status check programPosted:

Saki
  • Retired Staff
Status: Offline
Joined: Apr 09, 201113Year Member
Posts: 4,994
Reputation Power: 14230
Motto: Wow crazy USA hamburger yes
Motto: Wow crazy USA hamburger yes
Status: Offline
Joined: Apr 09, 201113Year Member
Posts: 4,994
Reputation Power: 14230
Motto: Wow crazy USA hamburger yes
Not sure how many of you know about this stuff or if you can help but I figured why not post here.


So I have this script, super simple. Powershell script that pings a URL to see if its alive or not. Everything works great, but the issue is I have 1000s of URLs and it takes quite a while to ping all them and receive a response. Talking 15-20 mins to do about 10,000 of them.

I'm sure there's a way to do this a little quicker. I've thought of running a few instances of this program at a time and "chunking" a url file of, lets say 10k and splitting it between 3 instances so each instance only has to fetch 3k~ URLs but they would be simultanious.

I honestly don't know much about programming and this is a very rudimentary setup I have currently.


Here's a pic of the code for reference:

https://i.imgur.com/Mc1tRWT.png



Any help is appreciated
#2. Posted:
0101010101010101
  • Spooky Poster
Status: Offline
Joined: Jul 15, 201212Year Member
Posts: 3,556
Reputation Power: 15228
Status: Offline
Joined: Jul 15, 201212Year Member
Posts: 3,556
Reputation Power: 15228
From past experience creating a loop and using a subset, let's say 100 at a time.
You will probably see faster results
#3. Posted:
Quarantine_Clean
  • Blind Luck
Status: Offline
Joined: Dec 02, 20204Year Member
Posts: 387
Reputation Power: 3040
Status: Offline
Joined: Dec 02, 20204Year Member
Posts: 387
Reputation Power: 3040
Will definitely be checking back for an answer. Ran into a similar problem when trying to compile a list of pc's to ping. Ended up just waiting the 1.5 hrs since I could not figure it out lmao
#4. Posted:
r00t
  • Administrator
Status: Offline
Joined: May 18, 201113Year Member
Posts: 16,419
Reputation Power: 24469
Status: Offline
Joined: May 18, 201113Year Member
Posts: 16,419
Reputation Power: 24469
Use Invoke-WebRequest and the UseBasicParsing switch. If you just need to see if the page errors, Invoke-RestMethod will be faster. Add $ProgressPreference = "SilentlyContinue".

PowerShell parallelism isn't great. Your options are using jobs, runspaces, and running multiple instances of the script.

devblogs.microsoft.com/scripting/...owershell/
devblogs.microsoft.com/scripting/...es-part-1/
#5. Posted:
CriticaI
  • Summer 2018
Status: Offline
Joined: Nov 05, 201311Year Member
Posts: 2,753
Reputation Power: 452
Status: Offline
Joined: Nov 05, 201311Year Member
Posts: 2,753
Reputation Power: 452
I have to ask, Why Powershell? Surely you can run something like Node, Python, PHP, or Ruby which has built-in support for making web requests?
Users browsing this topic: None
Jump to:


RECENT POSTS

HOT TOPICS