Upload size limit?

Developer
Nov 14, 2008 at 9:21 AM
I'm uploading files up to 4MB to scribd without a problem, but if I go to 5MB and above the files don't get uploaded.

I don't get any error messages. The files do get uploaded to the server, so it's not a web.config setting.

Is there some kind of limit imposed in this library or the sribd API?

cheers,

Steve
Coordinator
Nov 14, 2008 at 11:12 AM
Steve,

If there is a limit, it isn't imposed specifically by the client library so it may be an API limitation. 

What method are you using to upload?

- Paul
Developer
Nov 14, 2008 at 11:18 AM
There is no limit in the API because I mailed the Scribd developers directly.

Here's my code:

    protected void btnUpload_Click(object sender, EventArgs e)
    {
        try
        {
            if (loFile.PostedFile.ContentLength > 0)
            {
                loFile.PostedFile.SaveAs(Server.MapPath(PathToExistingFiles.Value) + "\\" + Path.GetFileName(loFile.PostedFile.FileName));
                // Set up upload notification
                Scribd.Net.Document.Uploaded += new EventHandler<DocumentEventArgs>(Document_Uploaded);

                string strPathToFile = Server.MapPath(PathToExistingFiles.Value) + "\\" + Path.GetFileName(loFile.PostedFile.FileName);
                string strDocType = Path.GetExtension(loFile.PostedFile.FileName).ToLower();
                strDocType = strDocType.Replace(".", "");
                // Send our PDF document to Scribd.
                try
                {
                    Scribd.Net.Document settings = Scribd.Net.Document.Upload(strPathToFile, strDocType);
                }
                catch (Exception Ex)
                {
                    lblError.Text = Ex.Message;
                }

                // now delete the file from the uploads folder
                if(File.Exists(strPathToFile))
                {
                    FileInfo fo = new FileInfo(strPathToFile);
                    fo.Delete();
                }
            }
        }
        catch (Exception Ex)
        {
            lblError.Text = Ex.Message;
        }
    }



    protected void Document_Uploaded(object sender, DocumentEventArgs e)
    {
        try
        {
            lblError.Text = "Uploaded " + e.Document.DocumentId;
        }
        catch (Exception Ex)
        {
            lblError.Text = Ex.Message;
        }
    }

I have this in web.config:

<httpRuntime useFullyQualifiedRedirectUrl="true" executionTimeout="7200" maxRequestLength="2048576" requestLengthDiskThreshold="2048576" />

It doesn't return any errors, just that the DocumentId = 0. I check directly at Scribd and the Doc isn't there.




Coordinator
Nov 14, 2008 at 11:40 AM
Hm......

I can't look at it right now, but I will later.  Feel free to look at the source code in the meantime.  It may be a time out issue or such with the way I'm sending it up to Scribd via the library.  I'm spitballing here, but maybe I can increase the chuck size for the upload.... I dunno.. I'll look at it.

- Paul
Developer
Nov 14, 2008 at 5:00 PM
I've had a look through the source code but nothing is jumping out at me.

I tried with several different 5MB pdfs but they all return DocumentId = 0, whereas smaller files are ok.

The files do get added to a temp folder on the server so it's not .NET that is limiting it.
Developer
Nov 16, 2008 at 9:48 AM
I realised that I could hook into the error handler like this:

Scribd.Net.Service.Error +=new EventHandler<ScribdEventArgs>(Service_Error);

    protected void Service_Error(object sender, ScribdEventArgs e)
    {
        lblError.Text = e.Message.ToString();
    }

This gave me the following message:
"System.Net.WebException: The request was aborted: The request was canceled."

When you look into this can you add this to the source and see if it works:
HttpWebRequest.KeepAlive = false;





cheers,
Steve


Developer
Nov 17, 2008 at 10:22 AM
Paul -

This might help: http://social.msdn.microsoft.com/Forums/en-US/netfxnetcom/thread/c904cc4c-fa9c-4e56-85fc-56a715cfafb3/

Steve
Coordinator
Nov 17, 2008 at 1:59 PM
Steve,

Thanks for all the information; it's really helpful.  It looks like I'm going to have to re-engineer the file upload functionality to fix this limitation... unless you're volunteering? :)

It appears to be the .Net webclient class that's the bottleneck - I'm not really doing anything special on the upload.  I guess a wrapper for HttpWebRequest is the way to go.

- Paul
Developer
Nov 17, 2008 at 3:21 PM
I'll gladly help you test it :)

I'm using this code in project that is pretty short on time, any ideas when you'll be able to fix this?
Coordinator
Nov 17, 2008 at 5:04 PM
I can probably get to this evening, however, I've made you a developer on the project so, y'know... feel free.... ;)
Coordinator
Nov 18, 2008 at 1:15 PM
Edited Nov 18, 2008 at 1:35 PM
Steve,

Try this:

In the Service.cs file, method: 

internal

Response PostRequest(Request request)

Replace the code:
byte [] _resp = _client.UploadFile(request.RESTCall, _fileName);

With this code:
byte [] _fileData = System.IO.File.ReadAllBytes(_fileName); 
byte[] _resp = _client.UploadData(request.RESTCall, _fileData);

- Paul

 

 

 

 

 

 

 

 

 

Developer
Nov 19, 2008 at 11:59 AM
As discussed by email, this didn't fix it.
Coordinator
Nov 19, 2008 at 1:31 PM
Yep.  I'm working on a solution now.

- Paul
Coordinator
Nov 19, 2008 at 3:57 PM
Steve,

I tried a few things out, and, after failing miserably, I decided to re-test the existing method.  I was able to upload a 6 MB file without a problem through the WinForms demo app.  That told me that it's not the library per-se. 

I did run into the problems you described with the web demo app, but I'm not sure what I can do about that.  It may have something to do with the webserver or something; I really don't know.

There's an overload to the Document.Upload method that accepts a Stream.  You may be able to leverage that.  However, keep in mind that that method will take the stream and write it to a temp file before using the traditional upload method.

Sorry I can't be of more help right now.

- Paul
Developer
Nov 21, 2008 at 7:58 PM
Thanks Paul, I'll give this a go.

My first test gave the result of 'Initial file saving failed' which tells me I need to set permissions somewhere for the temp file. Where is the temp file saved to?

cheers,

Steve
Developer
Nov 21, 2008 at 8:54 PM
Paul -

I tried your steam method as you had it in your tests web app:

Scribd.Net.Document.Upload(_fileStream, Scribd.Net.AccessTypes.Public);

This gave me the 'Initial file saving failed' error.

I switched it to pass an expicit file type like this:

Scribd.Net.Document settings = Scribd.Net.Document.Upload(_fileStream, "pdf");

This worked, but as soon as I tried a 5MB file, it bombed as before:

System.Net.WebException: The request was aborted: The request was canceled. at System.Net.WebClient.UploadFile(Uri address, String method, String fileName) at System.Net.WebClient.UploadFile(String address, String fileName) at Scribd.Net.Service.PostRequest(Request request)

I think the WebClient needs changing to HttpWebRequest to make this work :(





Developer
Nov 21, 2008 at 9:04 PM
Paul -

I don't know if this will help but I really need to get this working for at least 30MB files...

Anyway i did a bit more research and found this:

http://social.msdn.microsoft.com/Forums/en-US/netfxnetcom/thread/0d0afe40-c62a-4089-9d8b-fb4d206434dc/#page:2

Specifically, this post:

"I am posting this because we had a similar issue and solved it through a MSDN ticket.  A very knowledgeable engineer called me back and explained the issue.  This might help other people researching similar problems. 

The problem is solved by changing two values on the ServicePointManager class.  The first is the DefaultPersistentConnectionLimit field.  This has been addressed in numerous other posts.  It defaults to 2 which is too low for most intensive apps.  So bump this up.  We went to 100.  The next property is the MaxServicePointIdleTime property.  This sets the suggested time a connection is held after its last use before it is released back to the "queue."  By default this is 100 seconds.  This is way too high, the Microsoft engineer suggested 2000-5000 milliseconds for a highly concurrent application.  The combination of these two settings resolved all of our The request was aborted: The request was canceled exceptions.  The real key is the idle settings.

Many other posts refer to the Keepalive settings as the "work around" for this "bug."  Turning off Keepalive merely makes all connections immediately go back to the "queue."  So you are just avoiding the behavior that the Microsoft engineers designed into the framework.  Supposedly this was done to protect servers.  Hope this helps..."

I'm out of my depth with this, but I don't know if you can use these classes with your library - again, I'm happy to test it.

Steve




Coordinator
Nov 21, 2008 at 9:29 PM
The temp file is saved in whichever temp directory windows provides. I'll make this configurable in the next release.

Paul

-----Original Message-----
From: "SteveWhite" <[email removed]>
To: [email removed]
Sent: 11/21/08 3:59 PM
Subject: Re: Upload size limit? [scribdnet:39927]

From: SteveWhite

Thanks Paul, I'll give this a go.

My first test gave the result of 'Initial file saving failed' which tells me I need to set permissions somewhere for the temp file. Where is the temp file saved to?

cheers,

Steve


Coordinator
Nov 21, 2008 at 9:36 PM
Steve,

This is a limitation of the fileupload webcontrol. You can modify the upload size limit in the web.config file. The exact setting escapes me at the moment, but it's online somewhere..

Paul

-----Original Message-----
From: "SteveWhite" <[email removed]>
To: [email removed]
Sent: 11/21/08 4:54 PM
Subject: Re: Upload size limit? [scribdnet:39927]

From: SteveWhite

Paul -

I tried your steam method as you had it in your tests web app:

Scribd.Net.Document.Upload(_fileStream, Scribd.Net.AccessTypes.Public);

This gave me the 'Initial file saving failed' error.

I switched it to pass an expicit file type like this:

Scribd.Net.Document settings = Scribd.Net.Document.Upload(_fileStream, "pdf");

This worked, but as soon as I tried a 5MB file, it bombed as before:

System.Net.WebException: The request was aborted: The request was canceled.at System.Net.WebClient.UploadFile(Uri address, String method, String fileName)at System.Net.WebClient.UploadFile(String address, String fileName)at Scribd.Net.Service.PostRequest(Request request)

I think the WebClient needs changing to HttpWebRequest to make this work :(







Coordinator
Nov 21, 2008 at 11:04 PM
Steve,

I'll add this to the demo web site app, but here's what you add to the web.config to increase the size of files accepted by the asp upload control:

<

 

system.web>

 

<!--

 

Increase the maxRequestLength to accept larger files on upload. 51200 == 50 MB -->

 

<

 

httpRuntime

 

 

 

maxRequestLength="51200"

 

/>

Hope that helps -

Paul

Developer
Nov 22, 2008 at 10:02 AM
Paul -

See my 3rd post in this thread, I already added a huge amount to safeguard against this.

If I take out the scribd call, I can now upload 100MB+ files to the server no problem.

If I put the scribd call back in, it bombs with files over 4MB in the browser.

There is something that is limiting it getting to the scribd server, but it's not maxRequestLength and it's not the API.

It's more likely to be that it requires HttpWebRequest.KeepAlive = false;

Steve
Coordinator
Nov 22, 2008 at 2:26 PM
Steve,

Well, fudge.

Ok.. I'll implement the HttpWebRequest ...

bleh.

- Paul
Developer
Nov 25, 2008 at 8:27 PM
Thanks Paul, do you have any idea when this will be ready?

Steve
Developer
Dec 7, 2008 at 2:58 PM
This is now fixed in the latest version. The conversion to HttpWebRequest with KeepAlive="False" and setting the Request Timeout to System.Threading.Timeout.Infinite fixed.

This latest version now uploads up to 100MB files.