Author Topic: NANY 2011 Release: JottiQ (Read 263042 times)

worstje · « **Reply #100 on:** January 04, 2011, 02:07 PM »

Okay, I doubt the many people for which this is meant will read this, but it needs mentioning in case very slow results haven't made it obvious yet.

Just today, JottiQ has more than doubled the ALL-TIME HIGH queue of the internal scanning engine of Jotti's malware scan. Jotti has been running that since 2004. And obviously, this cannot stay this way, as it is way beyond his resources to manage.

I am currently in discussion with the man on ways to try and alleviate the problems, as I suspect they might be caused for a good part by the fact that NANY is a popular event with lots of people trying new applications. There is already rate-limiting present for an extent, but so far I have the impression Jotti believes this is not enough to handle the situation if it keeps on going.

Thus, possible ways he and/or I may respond are by adding load indicators to the JottiQ interface, automatic shutdown of Queue processing once stuff gets too crazy for him serverside, or as a final resort a full shutdown of JottiQ servicing so the service can remain for the diehards who would like to use the web interface.

So please, play nice with Jotti's server, especially till I get a new version out tomorrow. It's a free service, he wants to keep it that way, and I would truly hate for JottiQ to become a victim of its own success.

Ath · « **Reply #101 on:** January 04, 2011, 04:47 PM »

Hm, maybe somebody has some cpu-cycles on a linux host to spare, to relieve the load of the Jotti service? But then that Jotti service must be (made) capable of off-loading work to others ofcourse.

@worstje, does JottiQ add information about itself during the upload? In that case Jotti could put the requests on a lower priority (though that's not nice to the JottiQ users) but at least a temporary solution. And maybe you can now persuade him into accepting donations, so he can actually afford to buy more hardware...

An idea for improvement on the Jotti side (didn't see a forum over there) or maybe a JottiQ Quick Win (if not implemented yet...):
The 'Hash search' feature could be used on each entry, before a file is uploaded, because a database lookup should be much faster then 20 AV scanners to run across a file.

worstje · « **Reply #102 on:** January 04, 2011, 05:15 PM »

There's a fair few options open to Jotti. JottiQ has a unique API key it uses which at worst can be blocked all together, and the user agent that is sent includes the JottiQ version so that in case I make a huge blooper bug, he can block that specific version without affecting other versions.

Also, I need to clear something up that I've been meaning to clear up in the next release since I misunderstood Jotti at some point along my numerous emails. (I think we've sent over 100 emails at this point.) The service started out as a hobby project, but it hasn't been like that for a long time. There's a few parties that want some special features and they make up for some funding to keep the service running.

Sadly, as I was writing this reply, I got an email that at this time the load has skyrocketed some more to approximately 500% of its usual load. And sadly, the story regarding that remains the same: it is too much at this time to support. As such, Jotti has ceased service to JottiQ at present, hopefully to be reopened in upcoming days.

Jotti will monitor the usage and do several tests across the next few days to see if the demand will lessen. Personally, I am rather fearful JottiQ has garnered a bit too much attention for itself at present. Sites in Brazil, France, Poland and an english one have all done articles on JottiQ thus far, and somehow I expect it only to get worse. (Unless any more reviewers dump JottiQ because the service has now been shut down... hmm, if other methods to alleviate the pressure fail that might be the best I can hope for.)

kyrathaba · « **Reply #103 on:** January 04, 2011, 05:24 PM »

I surmise that the developer of Jotti will recognize the increased usefulness his program can provide in conjunction with JottiQ, and will likely be willing to work something out with you that will throttle the load. It just may take time.

Ath · « **Reply #104 on:** January 04, 2011, 05:35 PM »

Hm, bummer

Or to quote an older Dutch ad pay-off "Het waren 2 geweldige dagen" (It where 2 fabulous days)

I get a feeling this grew indeed a bit bigger than anyone, mostly Jotti IMHO, could have imagined, so he will probably have to restructure his service (or fix bugs in the Hash search, as that doesn't really seem to work), or get a bigger sponsor

Lest just pray he gets his act together, and JottiQ can provide it's useful service to the public again

The current errormessage of 'Jotti only supports files upto 0,00 B.' just says it all, I guess, maybe a next version of JottiQ should say 'Service temporarily unavailable' or something in that order? (But it shouldn't be needed I hope)

worstje · « **Reply #105 on:** January 04, 2011, 05:37 PM »

I hope so. It is mostly up to him - he is a busy person and I have zero access/say in his actual matters. I offered up a few alternatives for things I can change but right now it is just a matter of NANY 2011 catching the highlights and tons of people trying out JottiQ.

JottiQ was always meant for a bit of a poweruser in my eyes, to check over those few files your own AV package considered to be OK yet you still didn't trust. In case Jotti and in a lesser amount me are not able to find a satisfactory solution, I hope JottiQ simply falls out of favor... but well, let's just say I hope for the best but am secretly preparing for the worst case scenario.

Ath · « **Reply #106 on:** January 04, 2011, 05:39 PM »

Or you can rename JottiQ to VirusTotalQ, that would make a nice (and probably quick) NANY2012 pledge

mouser · « **Reply #107 on:** January 04, 2011, 05:42 PM »

I surmise that the developer of Jotti will recognize the increased usefulness his program can provide in conjunction with JottiQ, and will likely be willing to work something out with you that will throttle the load.

I expect and hope so too.

One thing JottiQ could do is insert a delay between uploading files, and a check of the jotti server status that increases that delay when the server is busy.

This really seems like a simple and full solution to the problem, even if its not ideal in the sense of making people wait longer than they should have to wait if the jotti service was a bit more sophisticated.

worstje · « **Reply #108 on:** January 04, 2011, 05:43 PM »

Or to quote an older Dutch ad pay-off "Het waren 2 geweldige dagen" (It where 2 fabulous days)
-Ath (January 04, 2011, 05:35 PM)

More than fabulous if you ask me.

I get a feeling this grew indeed a bit bigger than anyone, mostly Jotti IMHO, could have imagined, so he will probably have to restructure his service (or fix bugs in the Hash search, as that doesn't really seem to work), or get a bigger sponsor
Lest just pray he gets his act together, and JottiQ can provide it's useful service to the public again
-Ath (January 04, 2011, 05:35 PM)

I am not sure if the hash service is broken at present. For as far I can tell, he totally disabled the entire API at present, or at least for JottiQ's API credentials. I don't blame him - it's the simplest solution.

The current errormessage of 'Jotti only supports files upto 0,00 B.' just says it all, I guess, maybe a next version of JottiQ should say 'Service temporarily unavailable' or something in that order? (But it shouldn't be needed I hope)
-Ath (January 04, 2011, 05:35 PM)

It is ironic but true. The 0 bytes thing is what happens when the maximum file size cannot be retrieved, and in this case, it gives a strangely suitable message.

Or you can rename JottiQ to VirusTotalQ, that would make a nice (and probably quick) NANY2012 pledge
-Ath (January 04, 2011, 05:39 PM)

I have considered such a thing, actually. Supporting both services in one app might be nice, but I'd have to check up on VirusTotal, their requirements and all that other jazz. That and it would imply a ~~fairly big~~ huge architectural change.

worstje · « **Reply #109 on:** January 04, 2011, 05:45 PM »

One thing JottiQ could do is insert a delay between uploading files, and a check of the jotti server status that increases that delay when the server is busy.

This really seems like a simple and full solution to the problem, even if its not ideal in the sense of making people wait longer than they should have to wait if the jotti service was a bit more sophisticated.
-mouser (January 04, 2011, 05:42 PM)

There already is throttling. The problem isn't in people uploading a lot of stuff, it is in many people uploading stuff. Delays and all that sort won't change that. Suppose 500 people scan a single file. They all get allowed through. And that is what is clogging Jotti like mad right now. Everyone is downloading JottiQ, and everyone is throwing multiple files at it so by the time one file finally gets done the same person adds something new (although that should have gotten throttled by now.)

mouser · « **Reply #110 on:** January 04, 2011, 05:53 PM »

Well im not sure i buy the logic.

You are right that 500 people using JottiQ can result in 500 uploads to the service at a time, which if Jotti is not prepared for, could result in a heavy load on the server.

Now a pre-check for server status and a delay of N seconds before uploading when server is busy will definitely help.

BUT, even without checking the server status, the point is that it will make a HUGE additional difference in load if you delay between multiple file uploads, since most people using JottiQ will be queuing more than one file.

So it's all fine and well to say that adding a delay won't 100% solve every problem -- it is the case that delays will have a HUGE effect on lowering the load on the server.

I suggest a delay of something like max(uploads/2,10)*(0.5+serverload) (where serverload is from 0 to 1 based on server's reported serverload)

Adding the delay is something you have control over and could add really easily, and it would eliminate most of the problem and abuse.

worstje · « **Reply #111 on:** January 04, 2011, 06:02 PM »

Anything the delay can do is something Jotti can already do, and probably already does. JottiQ needs to get a token from the server in order to do anything for a particular 'file'. Said throttling is the delay you are aiming at that Jotti has FULL control over. If JottiQ doesn't get that token, it can do nothing.

I intend to add stuff to somehow involve the general server load, but again the ideal way is for the throttling to come from Jotti's side where possible. He has most control and insight into those things. His service can be at 100% load and not need to deny anyone. It is his internal queue of files people have uploaded that need scanning that is growing waay out of its intended capacity to the point where even the relatively quiet moments aren't providing enough space for him to minimize said internal queue to managable levels.

mouser · « **Reply #112 on:** January 04, 2011, 06:04 PM »

fair enough, jotti COULD ask your program to delay if he writes that code -- but if he can't get himself to write that in time, it still might be something you could do now to solve the problem.

worstje · « **Reply #113 on:** January 06, 2011, 05:56 AM »

JottiQ v1.0.2 is finally out. It is not the sort of release I would have preferred to make, but it is a necessity.

Changes in this version:

v1.0.2 (2011-01-06)

    JottiQ-got-too-popular-for-its-own-good release. At the highest point
    before Jotti's malware scan was forced to cease servicing JottiQ, the old
    all-time record for its scanning load had been beaten by over 500%.
    
      Added: Update check. It should have been implemented pre v1.0.0, but a
          lot of issues came up and it simply didn't happen. Have a look inside
          the 'About JottiQ' dialog. Future versions might add automation for
          this sort of checking. This supports the DCUpdater application.
      Added: A very heavy-handed approach to try and lighten the load on the
          servers. If the server passes a certain point, all non-cached files
          will not be uploaded and throw up an error. This way Jotti will
          hopefully no longer be forced to cease servicing JottiQ. In future
          versions, once JottiQ loses its spotlight, I will look into merely
          saving the actual file uploads for last in the case Jotti's malware
          scan is overloaded so one will not have to reset items to get stuff
          to scan once it quiets down a bit.
      Added: An actual link to the website and the discussion thread on
          DonationCoder. Forgetting to link to your own site is a bit of a
          stupid mistake to make.
      Added: Internet shortcuts. (Rather too many links than too little!)
      Added: Detection for withdrawn service from Jotti, and possible actions
          the user can take to alleviate the issue.
      Changed: Upgrading versions will require the 'Privacy Agreement' to be
          accepted once more.
      Changed: Users that put an unreasonably high load on Jotti's servers will
          from now on be reminded they are quite the inconsiderate users.
          I would rather not start truly limiting users as it will hurt the
          intended use-cases of the application as well, but if in the versions
          and months to come such soft reminders prove to not be enough, I will
          have to resort to limiting functionality in all sorts of ways.
      Changed: The older changelog (version history) entries have been split
          off into their own Changelog.txt file. This way the Readme will be a
          bit more accessible to users again.

mouser · « **Reply #114 on:** January 06, 2011, 10:54 AM »

it might be nice to show in the status or result window when a file cache was used instead of having to upload the whole file.
could be interesting to user and also useful in explaining why a file was so fast to be scanned.

worstje · « **Reply #115 on:** January 06, 2011, 11:02 AM »

I have been trying to hide the entire cache/non-cache thing for as much as possible. Once you start drawing a lot of attention to it, people will be all like 'oh my, I definitely need to re-scan since it might be a stale result'. There already is a second opinion feature for those users who truly need it, but in practice, it is really not necessary. The website itself also offers cached results even though you always upload the file in that case. Why? Because it is only rarely that you really need the file scanned again. The biggest issue is with 0-day viruses/worms and the sort which happened a lot in the days of the big viral email crap. Don't forget that JottiQ is not meant to be a first layer-of-defense - it is meant to be investigative in nature.

Also, while the item is processing you can keep an eye on the status message and/or icon in the queue to see what logical path is being taken. Uploading is very obvious, as are the 'file is queued' and 'intermediate results' bits.

Ath · « **Reply #116 on:** January 06, 2011, 05:20 PM »

It's really nice that Jotti opened the service again for JottiQ, congrats!

Now for all users to follow the presented guidelines on the intended-use-pattern...

worstje · « **Reply #117 on:** January 06, 2011, 05:46 PM »

So far, I think it helps greatly that v1.0.2 hasn't had much attention, and that v1.0.1 was getting most of its downloads from some major review sites with tons of users that all tried the app out at the same time. Sadly, v1.0.1 and prior had to be completely banned as a consequence.

At least from now on, JottiQ has an update-check, and it will be able to just error out once Jotti's general load becomes too heavy. It isn't an ideal solution, but given the timespan available to us it was among the best we could figure out. If all goes well over the next few months, I might be able to relax some things or Jotti might be able to get some more capacity. Time will have to tell.

kyrathaba · « **Reply #118 on:** January 07, 2011, 08:11 AM »

At any rate, even if neither of those things two things happens (and I think, personally, that one or the other -- perhaps both -- will), you've provided a very useful program for many people. Congrats!

Ashraf · « **Reply #119 on:** January 10, 2011, 03:26 AM »

Do you mind providing the portable version in .ZIP format? Not everyone can open .7z.

Ath · « **Reply #120 on:** January 10, 2011, 05:17 AM »

7-zip is a free download for Windows and Linux (but JottiQ still isn't going to work on Linux with Mono because of the WPF requirement)

mouser · « **Reply #121 on:** January 10, 2011, 08:35 AM »

JottiQ just got a long writeup with pictures at dottech.org:
http://dottech.org/f...eeware-reviews/21190

worstje · « **Reply #122 on:** January 10, 2011, 11:40 AM »

Do you mind providing the portable version in .ZIP format? Not everyone can open .7z.
-Ashraf (January 10, 2011, 03:26 AM)

I'll consider it once I wake up a bit more. The way I see it, .zip has been around since the 90ies, and while it is a good format, there's way better alternatives available already. I have not removed the requirement for the VC++ 2010 Redistributables because of efficiency and because as time passes, everyone should have those on their system. My logic is the same with the .7z format: everyone ought to get an archiver able to extract it nowadays.

Perhaps such 'progressive' thinking of mine is the wrong thing to use here, but if I don't try to push a better format a bit, who will? Big companies certainly won't as they have companies, users won't because older formats work fine for them, and so forth. It is a little thing to get a new archiver as even the builtin Windows support for .zip is the crappiest and slowest thing I've seen.

JottiQ just got a long writeup with pictures at dottech.org:
http://dottech.org/f...eeware-reviews/21190
-mouser (January 10, 2011, 08:35 AM)

Awesome, I'll go read it now.

worstje · « **Reply #123 on:** January 11, 2011, 08:08 PM »

Ok, so I admit it is addictive to look at the referers and notice all the articles and other nooks and crannies that link to JottiQ. And I noticed that a lot of them are foreign, non-english going through the effort of using and reviewing JottiQ. So I wonder.. maybe I should change that?

Let me know what you think.

Thanks.

ccondrup · « **Reply #124 on:** January 12, 2011, 04:48 PM »

Thanks for a nice frontend.

My two cents: I hadn't seen mention of the drag-drop part, so I just assumed there was an "add files"-button somewhere in the UI. One of the few enabled buttons was the "add running processes" so I assumed that was for adding files to queue and clicked it. That sparked 10+ scans which I hadn't planned on. Just saying - in case number of scans is still an issue - the current UI might lead to unneccesary scans.

Amusing that the only running process where a detection was reported was from Screenshotcaptor