MissingSchema - Invalid URL Error with Requests module - python-3.x

Trying to get data from the eBay API using GetItem. But requests isn't reading or getting the URL properly, any ideas? Getting this error:
requests.exceptions.MissingSchema: Invalid URL '<urllib.request.Request object at 0x00E86210>': No schema supplied. Perhaps you meant http://<urllib.request.Request object at 0x00E86210>?
I swear I had this code working before but now it's not, so I'm not sure why.
My code:
from urllib import request as urllib
import requests
url = 'https://api.ebay.com/ws/api.dll'
data = '''<?xml version="1.0" encoding="utf-8"?>
<GetItemRequest xmlns="urn:ebay:apis:eBLBaseComponents">
<RequesterCredentials>
<eBayAuthToken>my-auth-token</eBayAuthToken>
</RequesterCredentials>
<ItemID>any-item-id</ItemID>
</GetItemRequest>'''
headers = {
'Content-Type' : 'text/xml',
'X-EBAY-API-COMPATIBILITY-LEVEL' : 719,
'X-EBAY-API-DEV-NAME' : 'dev-name',
'X-EBAY-API-APP-NAME' : 'app-name',
'X-EBAY-API-CERT-NAME' : 'cert-name',
'X-EBAY-API-SITEID' : 3,
'X-EBAY-API-CALL-NAME' : 'GetItem',
}
req = urllib.Request(url, data, headers)
resp = requests.get(req)
content = resp.read()
print(content)
Thank you in advance. Any good reading material for urllib would be great, too.

You are mixing the urllib and the requests library. They are different libraries that can both do HTTP-requests in Python. I'd suggest you use only the requests library.
Remove the line req = urllib.Request(url, data, headers) and replace the resp = ... line with:
r = requests.post(url, data=data, headers=headers)
Print the response body like this:
print(r.text)
Check out the Requests Quickstart here for more examples: https://2.python-requests.org//en/master/user/quickstart/

Related

Appending XML body content in Python3 script

#!/usr/bin/python3
import requests
url = "https://test.test.com/api/dashboards
querystring = {'user' " 'bob'}
payload = "<?xml version =\"1.0\" encoding=\"UTF-8\"?>\r\n
the_remaining_body_pasted_from_Postman_snippet"
headers = {'Content-Type' : "application/xml"}
response = requests.request("POST", url, data=payload, headers=headers,
params=querystring, auth=('user', 'pass'))
print(response.text)
ResponseBodyParseException" name="Request Body Parse" statusCode="400"
An error occurred unmarshalling the document
Internal Exception: com.ctc.wstx.exc.WstxIOException: Invalid UTF-8 middle byte 0x6e (at char #3126, byte #37)
The call works within Postman but the snippet provided by Postman in Python3 does not seem to work pasted into .py file within Ubuntu.
Thank you

python3 urllib, make put request with no data

Hello I have a C# service, that takes a put request with no data, it just triggers an algorithm.
I am using urllib to make a request to this service, however I am getting a 411 error. This is because I am not supplying a data parameter.
req = urllib.request.Request(url, headers=headers, method='PUT')
try:
response = urllib.request.urlopen(req)
HTTPError: 411
If I try this:
req = urllib.request.Request(url, headers=headers,DATA="" method='PUT')
try:
response = urllib.request.urlopen(req)
TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str.
Can some one please help me, and please dont just say "huuu durr use requests" I cannot use imports. Thank you
Ok I solved this by:
req = urllib.request.Request(url, headers=headers,DATA=b'' method='PUT')
try:
response = urllib.request.urlopen(req)
Hope this helps

Create Google Shortened URLs, Update My CSV File

I've got a list of ~3,000 URLs I'm trying to create Google shortened links of, the idea is this CSV has a list of links and I want my code to output the shortened links in the column next to the original URLs.
I've been trying to modify the code found on this site here but I'm not skilled enough to get it to work.
Here's my code (I would not normally post an API key but the original person who asked this already posted it publicly on this site) :
import json
import pandas as pd
df = pd.read_csv('Links_Test.csv')
def shorternUrl(my_URL):
API_KEY = "AIzaSyCvhcU63u5OTnUsdYaCFtDkcutNm6lIEpw"
apiUrl = 'https://www.googleapis.com/urlshortener/v1/url'
longUrl = my_URL
headers = {"Content-type": "application/json"}
data = {"longUrl": longUrl}
h = httplib2.Http('.cache')
headers, response = h.request(apiUrl, "POST", json.dumps(data), headers)
return response
for url in df['URL']:
x = shorternUrl(url)
# Then I want it to write x into the column next to the original URL
But I only get errors at this point, before I even started figuring out how to write the new URLs to the CSV file.
Here's some sample data:
URL
www.apple.com
www.google.com
www.microsoft.com
www.linux.org
Thank you for your help,
Me
I think the issue is that you didnot include the API key in the request. By the way, the certifi package allows you to secure a connection to a link. You can get it using pip install certifi or pip urllib3[secure].
Here I create my own API key, so you might want to replace it with yours.
from urllib3 import PoolManager
import json
import certifi
sampleURL = 'http://www.apple.com'
APIkey = 'AIzaSyD8F41CL3nJBpEf0avqdQELKO2n962VXpA'
APIurl = 'https://www.googleapis.com/urlshortener/v1/url?key=' + APIkey
http = PoolManager(cert_reqs = 'CERT_REQUIRED', ca_certs=certifi.where())
def shortenURL(url):
data = {'key': APIkey, 'longUrl' : url}
response = http.request("POST", APIurl, body=json.dumps(data), headers= {'Content-Type' : 'application/json'}).data.decode('utf-8')
r = json.loads(response)
return (r['id'])
The decoding part converts the response object into a string so that we can convert it to a JSON and retrieve data.
From there on, you can store the data into another column and so on.
For the sampleUrl, I got back https(goo.gl/nujb) from the function.
I found a solution here:
https://pypi.python.org/pypi/pyshorteners
Example copied from linked page:
from pyshorteners import Shortener
url = 'http://www.google.com'
api_key = 'YOUR_API_KEY'
shortener = Shortener('Google', api_key=api_key)
print "My short url is {}".format(shortener.short(url))

upload file with post params

I want send file to url with some post params like "chat_id", "caption" and file must be sended as 'photo' field-name.
but I found only methods to send only POST params without file or sending only file without POST params and custom file name param..
This code send only params field, but I need also files field.
from urllib import parse, request
...
files={'photo': open('file.jpg','rb')}
params = parse.urlencode({'chat_id': chat_id, 'caption': 'test'})
#headers = {"Content-type": "multipart/form-data"}
req = request.Request(url, params.encode('ascii'))
response = request.urlopen(req)
print(response.read())
Python 3.4.2
ok. I found solution.
I install external lib Requests: HTTP for Humans.
and now i'm using requests
import requests
...
url = 'https://api.telegram.org/bot'+self.token+'/sendPhoto' #'/sendmessage'
print('request to '+url+' with params '+chat_id+' ' + text)
headers = {'Content-Type': 'multipart/form-data'}
files = {'photo': open('file.jpg', 'rb')}
params = {'chat_id': chat_id, 'caption': text}
r = requests.post(url, files=files, params=params)
print(r.text)

HTTPConnection.request fails but urllib.request.urlopen works?

I've been stuck here for quite a while but can't find anything helpful.
I'm trying to connect to a website and get a response json file in Python3. The code looks like below:
conn = http.client.HTTPConnection('host.address')
params = "xx"+ xx + "xx" + ...
conn.request('GET', '/a/b/c', params)
resp = conn.getresponse()
This actually won't return the json file but the webpage http://host.address/a/b/c, which is an error page.
However, while using the following code:
params = "xx"+ xx + "xx" + ...
resp = urllib.request.urlopen("http://host.address/a/b/c?"+params)
It returns correctly the json file.
Any idea what's wrong with the code?
Thanks
In HTTP, only POST requests are supposed to have a body. The third parameter to request() is actually the body (see http://docs.python.org/py3k/library/http.client.html#http.client.HTTPConnection.request) - just build the URL as shown in the second example.
Just to complement #sqrtsben's answer with an example:
import urllib.parse
import http.client
u = urllib.parse.urlparse("http://localhost:8080/index.php?utf8=✓")
conn = http.client.HTTPConnection(u.hostname, u.port)
if u.query == '':
conn.request("GET", u.path)
else:
conn.request("GET", u.path + '?' + u.query)

Resources