python - Simulating HTTP POST request produces unexpected result -

---solved---

it turned out request body had literal r"\n" (repr: "\\n") characters in it, , since copy pasted body python string, python thought giving newline characters rather escaped newline characters.

the reason causes bad request follows: body json, , in json have escape newline characters definition. when server loads json object raw text, error thrown causing bad request

i realised because content-length header different in both cases (\n 1 char while \\\n 2 chars, although perhaps content-length doesn't matter.

also noteworthy when lower content-length sent, bad request returned. believe because json body gets truncated, , server doesn't accept important char (e.g. closing brace or something)

--- problem:---

summary:

i trying use python simulate post request bitbucket.org performed within firefox web browser. here did:

1. tracked post request using firebug
2. copied post request headers
3. copied post request body (in application/json format)

code:

here code use post request, it's bit long , not relevant. content-type application/json, , post body json-encoded string.

    datastring = '{"branch":"master","files":[{"path":"readme.txt","content":"ntestxx\n \n"}],"message":"readme.txt edited online bitbucket","parents":["465305dc4da32f91da057b65297cda9b72c"],"repository":{"full_name":"minesite/ica-i18n"},"timestamp":"2014-03-20t23:49:29.759z","transient":false}'     headers = {'x-csrftoken': '6tqwjcl698u99iu6zygbalocxz', 'content-length': '2190', 'accept-language': 'en,en-us;q=0.7,zh;q=0.3', 'x-newrelic-id': 'vwmgvvzsgwiiufbqdwu=, vwmgvvzsgwiiufbqdwu=', 'cookie': 'csrftoken=6tqwjcl698u99iu6zygbalocxz; __utma=254090395.1171276563.1394767875.1394776803.1395358874.3; __utmc=254090395; __utmz=254090395.1394776803.2.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); bb_session=gpqergylgoa7icpwosqsbpxig0; __utmv=254090395.|1=isbbuser=true=1; recently-viewed-repos_1701252=3802872%2c108928; __utmb=254090395.21.9.1395359363952', 'connection': 'keep-alive', 'accept': 'application/json, text/javascript, */*; q=0.01', 'user-agent': 'mozilla/5.0 (x11; ubuntu; linux x86_64; rv:27.0) gecko/20100101 firefox/27.0', 'host': 'bitbucket.org', 'x-requested-with': 'xmlhttprequest', 'pragma': 'no-cache', 'cache-control': 'no-cache', 'referer': 'https://bitbucket.org/xxxxxx/xxxxxxx/src/465305dc4da32f91da057b6529a8e4/readme.txt?at=master', 'content-type': 'application/json; charset=utf-8', 'accept-encoding': 'gzip, deflate'}     edit = requests.post("https://bitbucket.org/!api/internal/repositories/xxxxxxx/xxxxxxxx/oecommits/", data=datastring, headers=headers) 

results vs. expected results:

when perform post request using firefox web browser (using firebug's "resend request" function), 409 conflict response (which desired response! simulating request online editor, should correct response re-sent edit).

however, when try simulate request copying request header , request body, 400 bad request response, , response contains no other information, don't know problem is.

regardless of how many times send post in web-browser (despite incorrect timestamp), achieves intended outcome, server refuses accept requests make using python requests library.

response using browser request:

headers

http/1.1 409 conflict server: nginx/1.5.10 date: fri, 21 mar 2014 00:20:55 gmt content-type: text/plain content-length: 45 connection: keep-alive x-served-by: app16 x-render-time: 0.558492183685 content-language: en x-static-version: 48695e7c3140 vary: authorization, accept-language, cookie x-version: e6778a5040f7 etag: "92f0b780984e984140de0f8ed0a3992c" x-frame-options: sameorigin x-request-count: 483 x-newrelic-app-data: pxqevfdxcaitvvlwbgmpukydfgqhbdcquqxla1tmxv1dsn8uxwjhcwtygampf1pguw8efhlqrxyxh1ddc0gkdeqhsgxzvbpaugtddvqtqfgrwfsicaz9v1kqig1axf4slfbyvw4deuxtef0dtf0whgnjcu8evapuuguhvffqcgqcu1fxgwmgx1qdfaebuvvba1ajvqebb1fsa11dhqdsdhdtag== 

body

specified change not on head of branch master 

response using python request:

headers

content-length: 11 x-served-by: app10 x-render-time: 0.012787103653 content-language: en content-type: text/plain vary: authorization, accept-language, cookie connection: keep-alive server: nginx/1.5.10 x-version: e6778a5040f7 etag: "825644f747baab2c00e420dbbc39e4b3" x-request-count: 321 x-newrelic-app-data: pxqevfdxcaitvvlwbgmpukydfgqhbdcquqxla1tmxv1dsn8uxwjhcwtygampf1pguw8efhlqrxyxh1ddc0grb0mntrbbxq5gvhzwfemcvkbibhtrsfmjaarqulsdbw9vxaibc1twvu4cuwtufbpvawfcwgdtvqiaxqbrwqqagh9wbq0rumw= date: fri, 21 mar 2014 00:51:01 gmt x-frame-options: sameorigin x-static-version: 48695e7c3140 

body

bad request 

some of ideas:

i thinking perhaps there component http post request need simulate? perhaps when firefox sends post request, there header or wrapper added makes request valid?

or there more post request method, headers, , body?

maybe it's fact it's https instead of http?

update:

i have tried sending "sent cookies" in request, little success.

or there more post request method, headers, , body?

no. important part request headers. should same in both cases.

because firebug can track network requests inside firefox, you'll need external network analyzer wireshark track requests coming python script. of course need run on server script lies.

another solution run request against local web server , log request information there.

then you'll able compare request made in browser 1 script.