topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Sunday October 17, 2021, 4:44 pm
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: Grabbing files from authenticated website  (Read 1514 times)

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,768
    • View Profile
    • Donate to Member
Grabbing files from authenticated website
« on: September 23, 2021, 05:18 AM »
Hello

I want to grab xls files via specific links through a website that has a form authentication. Because it has an additional sms authentication step, it will not be possible to fully automate a live connection.

The session is set at 60 minutes so I am thinking if I ping/query the website every 30min, the cookie will renew/refresh.

However, this pinging will need to come from a 24/7 server and not my laptop as I cannot have it on all the time.

Is there any free solution for this?

I was thinking to create a Sharepoint Flow where it will download the xls every 30 min refreshing the cookie/auth/sessionID each time. I was hoping that I can somehow login manually for the first time via Sharepoint and then since the flow will be scheduled every 30, it will run forever refreshing the session.

Or do you have any other idea?
I was checking some of the free Azure services if they are suitable but no luck.

Thanks!

publicdomain

  • Honorary Member
  • Joined in 2019
  • **
  • Posts: 322
  • Call me Vic!
    • View Profile
    • PublicDomainWeekly.com
    • Read more about this member.
    • Donate to Member
Re: Grabbing files from authenticated website
« Reply #1 on: September 23, 2021, 10:27 AM »
... that has a form authentication.

Hello Kalos! What's the URL/Link to the authentication form? Chances are it can be automated via a browser automation script + a GUI.

Cheers!
✔️ New releases: PublicDomainWeekly.com

🎁 One-time support: Paypal @ www.paypal.com/paypalme/victorvls

❤️ Steady support: Patreon @ www.patreon.com/publicdomain

Thanks & keep on rocking!
Email/Paypal: publicdomainvicgmail.com

Coding: GingerTG v0.1.0
--Victor
« Last Edit: September 23, 2021, 11:26 AM by publicdomain »

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,768
    • View Profile
    • Donate to Member
Re: Grabbing files from authenticated website
« Reply #2 on: September 23, 2021, 03:02 PM »
Hi!

Unfortunately it's a proprietary site and I cannot tell it.

It is a form authentication website which has an additional step of authentication so it is really really hard to automate it.

Instead, I am just looking to download specific urls every 30 min through the IE/Edge so that the session does not expire.

What would be the best way to do that? I tried Powershell, Windows Task Scheduler, Excel Data Connections but non really worked due to downloading either nothing or the login page.

publicdomain

  • Honorary Member
  • Joined in 2019
  • **
  • Posts: 322
  • Call me Vic!
    • View Profile
    • PublicDomainWeekly.com
    • Read more about this member.
    • Donate to Member
Re: Grabbing files from authenticated website
« Reply #3 on: September 23, 2021, 03:45 PM »
Unfortunately it's a proprietary site and I cannot tell it.

No problem 8)

What would be the best way to do that?

I would either use Selenium (https://www.selenium.dev) or a simple "embedded" WebView/WebBrowser for this.

If you're in a rush, we can discuss privately, no biggie. As long as the resulting program goes to serve the public community, I'm in!  :)

(BTW, no hard-coded private URLs, just configurable fields)
✔️ New releases: PublicDomainWeekly.com

🎁 One-time support: Paypal @ www.paypal.com/paypalme/victorvls

❤️ Steady support: Patreon @ www.patreon.com/publicdomain

Thanks & keep on rocking!
Email/Paypal: publicdomainvicgmail.com

Coding: GingerTG v0.1.0
--Victor

Shades

  • Member
  • Joined in 2006
  • **
  • Posts: 2,848
    • View Profile
    • Donate to Member
Re: Grabbing files from authenticated website
« Reply #4 on: September 23, 2021, 08:20 PM »
Hello

I want to grab xls files via specific links through a website that has a form authentication. Because it has an additional sms authentication step, it will not be possible to fully automate a live connection.

The session is set at 60 minutes so I am thinking if I ping/query the website every 30min, the cookie will renew/refresh.

However, this pinging will need to come from a 24/7 server and not my laptop as I cannot have it on all the time.

Is there any free solution for this?

I was thinking to create a Sharepoint Flow where it will download the xls every 30 min refreshing the cookie/auth/sessionID each time. I was hoping that I can somehow login manually for the first time via Sharepoint and then since the flow will be scheduled every 30, it will run forever refreshing the session.

Or do you have any other idea?
I was checking some of the free Azure services if they are suitable but no luck.

Thanks!

Although it isn't a safe practice, you can add your credentials directly into the URL of the site you wish to visit.

Like so:
https://<user name>:<password>@<website URL>

or in a example:
https://kalos:[email protected]

If needed, you can even add a different port number:   https://kalos:[email protected]:8080