Commit e8a2f088 authored by John Chen's avatar John Chen Committed by Commit Bot

[ChromeDriver] Remove waterfall related code

Clean up code that has become obsolete due to decomission of
ChromeDriver waterfall.

Change-Id: Ifbdb43872111ec3e575ec6e4aa1b28b76b663756
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1509095
Commit-Queue: John Chen <johnchen@chromium.org>
Reviewed-by: default avatarCaleb Rouleau <crouleau@chromium.org>
Cr-Commit-Position: refs/heads/master@{#638687}
parent f1181050
This file contains high-level info about how ChromeDriver works and how to This file contains high-level info about how ChromeDriver works and how to
contribute. contribute.
ChromeDriver is an implementation of the WebDriver standard, ChromeDriver is an implementation of the WebDriver standard
which allows users to automate testing of their website across browsers. (https://w3c.github.io/webdriver/), which allows users to automate testing of
their website across browsers.
See the user site at https://sites.google.com/a/chromium.org/chromedriver/ See the user site at https://sites.google.com/a/chromium.org/chromedriver/
...@@ -11,7 +12,10 @@ Build ChromeDriver by building the 'chromedriver' target. This will ...@@ -11,7 +12,10 @@ Build ChromeDriver by building the 'chromedriver' target. This will
create an executable binary in the build folder named create an executable binary in the build folder named
'chromedriver[.exe]'. 'chromedriver[.exe]'.
Once built, ChromeDriver can be used interactively with python. Once built, ChromeDriver can be used with various third-party libraries that
support WebDriver protocol, including language bindings provided by Selenium.
For testing purposes, ChromeDriver can be used interactively with python.
$ export PYTHONPATH=<THIS_DIR>/server:<THIS_DIR>/client $ export PYTHONPATH=<THIS_DIR>/server:<THIS_DIR>/client
$ python $ python
...@@ -25,7 +29,7 @@ $ python ...@@ -25,7 +29,7 @@ $ python
ChromeDriver will use the system installed Chrome by default. ChromeDriver will use the system installed Chrome by default.
To use ChromeDriver2 with Chrome on Android pass the Android package name in the To use ChromeDriver with Chrome on Android pass the Android package name in the
chromeOptions.androidPackage capability when creating the driver. The path to chromeOptions.androidPackage capability when creating the driver. The path to
adb_commands.py and the adb tool from the Android SDK must be set in PATH. For adb_commands.py and the adb tool from the Android SDK must be set in PATH. For
more detailed instructions see the user site. more detailed instructions see the user site.
...@@ -79,30 +83,25 @@ Integration tests. ...@@ -79,30 +83,25 @@ Integration tests.
Third party libraries used by chromedriver. Third party libraries used by chromedriver.
=====Testing===== =====Testing=====
See the ChromeDriver waterfall at: There are several test suites for verifying ChromeDriver's correctness:
http://build.chromium.org/p/chromium.chromedriver/waterfall
There are 4 test suites for verifying ChromeDriver's correctness:
1) chromedriver_unittests (chrome/chrome_tests.gypi)
This is the unittest target, which runs on the main waterfall on win/mac/linux
and can close the tree. It is also run on the commit queue and try bots by
default. Tests should take a few milliseconds and be very stable.
2) chromedriver_tests (chrome/chrome_tests.gypi) * chromedriver_unittests
This is a collection of C++ medium sized tests which can be run optionally This is the unittest target, which runs on the commit queue on win/mac/linux.
on the trybots. Tests should take a few milliseconds and be very stable.
3) python integration tests * python integration tests (test/run_py_tests.py)
Run test/run_py_tests.py --help for more info. These are only run on the These tests are maintained by the ChromeDriver team, and are intended to verify
ChromeDriver waterfall. that ChromeDriver works correctly with Chrome. Run test/run_py_tests.py --help
for more info. These are run on the commit queue on win/mac/linux.
4) WebDriver Java acceptance tests * WebDriver Java acceptance tests (test/run_java_tests.py)
These are integration tests from the WebDriver open source project which can These are integration tests from the WebDriver open source project which can
be run via test/run_java_tests.py. They are only run on the ChromeDriver be run via test/run_java_tests.py. They are not currently run on any bots, but
bots. Run with --help for more info. will be included in the commit queue in the future. Run with --help for more
info.
=====Contributing===== =====Contributing=====
Find an open issue and submit a patch for review by an individual listed in Find an open issue and submit a patch for review by an individual listed in
the OWNERS file in this directory. Issues are tracked in chromedriver's issue the OWNERS file in this directory. Issues are tracked in chromedriver's issue
tracker: tracker:
https://code.google.com/p/chromedriver/issues/list https://bugs.chromium.org/p/chromedriver/issues/list
# Copyright (c) 2013 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
"""Downloads items from the Chromium snapshot archive."""
import json
import os
import re
import urllib
import urllib2
import util
_SITE = 'http://commondatastorage.googleapis.com'
GS_GIT_LOG_URL = (
'https://chromium.googlesource.com/chromium/src/+/%s?format=json')
GS_SEARCH_PATTERN = r'Cr-Commit-Position: refs/heads/master@{#(\d+)}'
CR_REV_URL = 'https://cr-rev.appspot.com/_ah/api/crrev/v1/redirect/%s'
class Site(object):
CHROMIUM_SNAPSHOT = _SITE + '/chromium-browser-snapshots'
def GetLatestRevision():
"""Returns the latest revision (as a string) available for this platform.
Args:
site: the archive site to check against, default to the snapshot one.
"""
url = '%s/%s/LAST_CHANGE' % (GetDownloadSite(), _GetDownloadPlatform())
return urllib.urlopen(url).read()
def DownloadChrome(revision, dest_dir, site=Site.CHROMIUM_SNAPSHOT):
"""Downloads the packaged Chrome from the archive to the given directory.
Args:
revision: the revision of Chrome to download.
dest_dir: the directory to download Chrome to.
site: the archive site to download from, default to the snapshot one.
Returns:
The path to the unzipped Chrome binary.
"""
def GetZipName(revision):
if util.IsWindows():
return revision + (
'/chrome-win.zip' if int(revision) > 591478 else '/chrome-win32.zip')
elif util.IsMac():
return revision + '/chrome-mac.zip'
elif util.IsLinux():
return revision + '/chrome-linux.zip'
def GetDirName(revision):
if util.IsWindows():
return 'chrome-win' if int(revision) > 591478 else 'chrome-win32'
elif util.IsMac():
return 'chrome-mac'
elif util.IsLinux():
return 'chrome-linux'
def GetChromePathFromPackage():
if util.IsWindows():
return 'chrome.exe'
elif util.IsMac():
return 'Chromium.app/Contents/MacOS/Chromium'
elif util.IsLinux():
return 'chrome-wrapper'
zip_path = os.path.join(dest_dir, 'chrome-%s.zip' % revision)
if not os.path.exists(zip_path):
url = site + '/%s/%s' % (_GetDownloadPlatform(), GetZipName(revision))
print 'Downloading', url, '...'
urllib.urlretrieve(url, zip_path)
util.Unzip(zip_path, dest_dir)
return os.path.join(dest_dir, GetDirName(revision),
GetChromePathFromPackage())
def _GetDownloadPlatform():
"""Returns the name for this platform on the archive site."""
if util.IsWindows():
return 'Win'
elif util.IsMac():
return 'Mac'
elif util.IsLinux():
return 'Linux_x64'
def GetLatestSnapshotPosition():
"""Returns the latest commit position of snapshot build."""
latest_revision = GetLatestRevision()
return latest_revision
def GetDownloadSite():
"""Returns the site to download snapshot build according to the platform."""
return Site.CHROMIUM_SNAPSHOT
def GetCommitPositionFromGitHash(snapshot_hashcode):
json_url = GS_GIT_LOG_URL % snapshot_hashcode
try:
response = urllib2.urlopen(json_url)
except urllib2.HTTPError as error:
util.PrintAndFlush('HTTP Error %d' % error.getcode())
return None
except urllib2.URLError as error:
util.PrintAndFlush('URL Error %s' % error.message)
return None
data = json.loads(response.read()[4:])
if 'message' in data:
message = data['message'].split('\n')
message = [line for line in message if line.strip()]
search_pattern = re.compile(GS_SEARCH_PATTERN)
result = search_pattern.search(message[len(message)-1])
if result:
return result.group(1)
util.PrintAndFlush('Failed to get commit position number for %s' %
snapshot_hashcode)
return None
def _GetGitHashFromCommitPosition(commit_position):
json_url = CR_REV_URL % commit_position
try:
response = urllib2.urlopen(json_url)
except urllib2.HTTPError as error:
util.PrintAndFlush('HTTP Error %d' % error.getcode())
return None
except urllib2.URLError as error:
util.PrintAndFlush('URL Error %s' % error.message)
return None
data = json.loads(response.read())
if 'git_sha' in data:
return data['git_sha']
util.PrintAndFlush('Failed to get git hash for %s' % commit_position)
return None
def _GetFirstBuildAfterBranch(branch_position):
latest_revision = GetLatestSnapshotPosition()
for commit_position in range(int(branch_position), int(latest_revision)):
git_hash = _GetGitHashFromCommitPosition(commit_position)
try:
_ = DownloadChrome(git_hash, util.MakeTempDir(), GetDownloadSite())
return git_hash
except:
continue
return None
This diff is collapsed.
#!/usr/bin/env python
# Copyright 2013 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
"""Runs all ChromeDriver end to end tests."""
import optparse
import os
import platform
import shutil
import sys
import tempfile
import traceback
_THIS_DIR = os.path.abspath(os.path.dirname(__file__))
_PARENT_DIR = os.path.join(_THIS_DIR, os.pardir)
sys.path.insert(0, _PARENT_DIR)
import archive
import chrome_paths
import util
sys.path.insert(0, os.path.join(chrome_paths.GetSrc(), 'build', 'android'))
from pylib import constants
def _GenerateTestCommand(script,
chromedriver,
chrome=None,
android_package=None,
verbose=False):
_, log_path = tempfile.mkstemp(prefix='chromedriver_log_')
print 'chromedriver server log: %s' % log_path
cmd = [
sys.executable,
os.path.join(_THIS_DIR, script),
'--chromedriver=%s' % chromedriver,
'--log-path=%s' % log_path,
]
if chrome:
cmd.append('--chrome=' + chrome)
if verbose:
cmd.append('--verbose')
if android_package:
cmd = ['xvfb-run', '-a'] + cmd
cmd.append('--android-package=' + android_package)
return cmd
def RunReplayTests(chromedriver, chrome):
util.MarkBuildStepStart('replay_tests')
_, log_path = tempfile.mkstemp(prefix='chromedriver_log_')
print 'chromedriver server log: %s' % log_path
cmd = [
sys.executable,
os.path.join(_PARENT_DIR, 'log_replay', 'client_replay_test.py'),
chromedriver,
chrome,
'--output-log-path=%s' % log_path
]
code = util.RunCommand(cmd)
if code:
util.MarkBuildStepError()
return code
def RunPythonTests(chromedriver,
chrome=None,
android_package=None):
util.MarkBuildStepStart('python_tests')
code = util.RunCommand(
_GenerateTestCommand('run_py_tests.py',
chromedriver,
chrome=chrome,
android_package=android_package))
if code:
util.MarkBuildStepError()
return code
def RunJavaTests(chromedriver, chrome=None,
android_package=None,
verbose=False):
util.MarkBuildStepStart('java_tests')
code = util.RunCommand(
_GenerateTestCommand('run_java_tests.py',
chromedriver,
chrome=chrome,
android_package=android_package,
verbose=verbose))
if code:
util.MarkBuildStepError()
return code
def RunCppTests(cpp_tests):
util.MarkBuildStepStart('chromedriver_tests')
code = util.RunCommand([cpp_tests])
if code:
util.MarkBuildStepError()
return code
def DownloadChrome(version_name, revision, download_site):
util.MarkBuildStepStart('download %s' % version_name)
try:
temp_dir = util.MakeTempDir()
return (temp_dir, archive.DownloadChrome(revision, temp_dir, download_site))
except Exception:
traceback.print_exc()
util.AddBuildStepText('Skip Java and Python tests')
util.MarkBuildStepError()
return (None, None)
def _KillChromes():
chrome_map = {
'win': 'chrome.exe',
'mac': 'Chromium',
'linux': 'chrome',
}
if util.IsWindows():
cmd = ['taskkill', '/F', '/IM']
else:
cmd = ['killall', '-9']
cmd.append(chrome_map[util.GetPlatformName()])
util.RunCommand(cmd)
def main():
parser = optparse.OptionParser()
parser.add_option(
'', '--android-packages',
help='Comma separated list of application package names, '
'if running tests on Android.')
options, _ = parser.parse_args()
exe_postfix = ''
if util.IsWindows():
exe_postfix = '.exe'
cpp_tests_name = 'chromedriver_tests' + exe_postfix
server_name = 'chromedriver' + exe_postfix
required_build_outputs = [server_name]
if not options.android_packages:
required_build_outputs += [cpp_tests_name]
try:
build_dir = chrome_paths.GetBuildDir(required_build_outputs)
except RuntimeError:
util.MarkBuildStepStart('check required binaries')
traceback.print_exc()
util.MarkBuildStepError()
constants.SetBuildType(os.path.basename(build_dir))
print 'Using build outputs from', build_dir
chromedriver = os.path.join(build_dir, server_name)
platform_name = util.GetPlatformName()
if util.IsLinux() and util.Is64Bit():
platform_name += '64'
if options.android_packages:
os.environ['PATH'] += os.pathsep + os.path.join(
_THIS_DIR, os.pardir, 'chrome')
code = 0
for package in options.android_packages.split(','):
code1 = RunPythonTests(chromedriver,
chrome_version_name=package,
android_package=package)
code2 = RunJavaTests(chromedriver,
chrome_version_name=package,
android_package=package,
verbose=True)
code = code or code1 or code2
return code
else:
code = 0
download_site = archive.GetDownloadSite()
revision = archive.GetLatestRevision()
temp_dir, chrome_path = DownloadChrome(revision, revision,
download_site)
if not chrome_path:
code = 1
code1 = RunPythonTests(chromedriver,
chrome=chrome_path)
code2 = RunJavaTests(chromedriver,
verbose=True,
chrome=chrome_path)
code3 = RunReplayTests(chromedriver,
chrome=chrome_path)
code = code or code1 or code2 or code3
_KillChromes()
shutil.rmtree(temp_dir)
cpp_tests = os.path.join(build_dir, cpp_tests_name)
return RunCppTests(cpp_tests) or code
if __name__ == '__main__':
sys.exit(main())
...@@ -3,13 +3,7 @@ ...@@ -3,13 +3,7 @@
# Use of this source code is governed by a BSD-style license that can be # Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file. # found in the LICENSE file.
"""Runs the WebDriver Java acceptance tests. """Runs the WebDriver Java acceptance tests."""
This script is called from chrome/test/chromedriver/run_all_tests.py and reports
results using the buildbot annotation scheme.
For ChromeDriver documentation, refer to http://code.google.com/p/chromedriver.
"""
import optparse import optparse
import os import os
......
{
"build_info": {
"win7" : {
"builder_url" : "http://build.chromium.org/p/chromium.chromedriver/builders/Win7/builds/",
"json_url" : "http://build.chromium.org/p/chromium.chromedriver/json/builders/Win7/builds/%d?as_text=1",
"builds_url": "http://build.chromium.org/p/chromium.chromedriver/json/builders/Win7/builds/_all?as_text=1"
},
"mac_10_6" : {
"builder_url" : "http://build.chromium.org/p/chromium.chromedriver/builders/Mac%2010.6/builds/",
"json_url" : "http://build.chromium.org/p/chromium.chromedriver/json/builders/Mac 10.6/builds/%d?as_text=1",
"builds_url": "http://build.chromium.org/p/chromium.chromedriver/json/builders/Mac 10.6/builds/_all?as_text=1"
},
"linux" : {
"builder_url" : "http://build.chromium.org/p/chromium.chromedriver/builders/Linux/builds/",
"json_url" : "http://build.chromium.org/p/chromium.chromedriver/json/builders/Linux/builds/%d?as_text=1",
"builds_url": "http://build.chromium.org/p/chromium.chromedriver/json/builders/Linux/builds/_all?as_text=1"
},
"linux32" : {
"builder_url" : "http://build.chromium.org/p/chromium.chromedriver/builders/Linux32/builds/",
"json_url" : "http://build.chromium.org/p/chromium.chromedriver/json/builders/Linux32/builds/%d?as_text=1",
"builds_url": "http://build.chromium.org/p/chromium.chromedriver/json/builders/Linux32/builds/_all?as_text=1"
},
"android_chromedriver_test" : {
"builder_url" : "http://build.chromium.org/p/chromium.fyi/builders/Android%20ChromeDriver%20Tests%20%28dbg%29/builds/",
"json_url" : "http://build.chromium.org/p/chromium.fyi/json/builders/Android ChromeDriver Tests (dbg)/builds/%d?as_text=1",
"builds_url": "http://build.chromium.org/p/chromium.fyi/json/builders/Android ChromeDriver Tests (dbg)/builds/_all?as_text=1"
}
},
"recipient_emails": "pshenoy@chromium.org",
"sender_email": "pshenoy@chromium.org",
"old_build_days": 2
}
#!/usr/bin/env python
# Copyright (c) 2014 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
"""Waterfall monitoring script.
This script checks all builders specified in the config file and sends
status email about any step failures in these builders. This also
reports a build as failure if the latest build on that builder was built
2 days back. (Number of days can be configured in the config file)
This script can be run as cronjob on a linux machine once a day and
get email notification for any waterfall specified in the config file.
Sample cronjob entry below. This entry will run the script everyday at 9 AM.
Include this in the crontab file.
0 9 * * * <Path to script> --config <Path to json file>
"""
import datetime
import json
import optparse
import sys
import time
import traceback
import urllib
from datetime import timedelta
from email.mime.text import MIMEText
from subprocess import Popen, PIPE
SUCCESS_SUBJECT = ('[CHROME TESTING]: Builder status %s: PASSED.')
FAILURE_SUBJECT = ('[CHROME TESTING]: Builder status %s: FAILED %d out of %d')
EXCEPTION_SUBJECT = ('Exception occurred running waterfall_builder_monitor.py '
'script')
def GetTimeDelta(date, days):
if isinstance(date, datetime.datetime):
return date + timedelta(days)
def GetDateFromEpochFormat(epoch_time):
last_build_date = time.localtime(epoch_time)
last_build_date = datetime.datetime(int(last_build_date.tm_year),
int(last_build_date.tm_mon),
int(last_build_date.tm_mday),
int(last_build_date.tm_hour),
int(last_build_date.tm_min),
int(last_build_date.tm_sec))
return last_build_date
def GetJSONData(json_url):
response = urllib.urlopen(json_url)
if response.getcode() == 200:
try:
data = json.loads(response.read())
except ValueError:
print 'ValueError for JSON URL: %s' % json_url
raise
else:
raise Exception('Error from URL: %s' % json_url)
response.close()
return data
def SendEmailViaSendmailCommand(sender_email, recipient_emails,
subject, email_body):
msg = MIMEText(email_body)
msg["From"] = sender_email
msg["To"] = recipient_emails
msg["Subject"] = subject
pipe = Popen(["/usr/sbin/sendmail", "-t"], stdin=PIPE)
pipe.communicate(msg.as_string())
def SendStatusEmailViaSendmailCommand(consolidated_results,
recipient_emails,
sender_email):
failure_count = 0
for result in consolidated_results:
if result['error'] != 'passed' and not result['build_too_old']:
failure_count += 1
today = str(datetime.date.today()).replace('-', '/')[5:]
if failure_count == 0:
subject = SUCCESS_SUBJECT % today
else:
subject = FAILURE_SUBJECT % (today,
failure_count,
len(consolidated_results))
email_body = ''
for result in consolidated_results:
if result['error'] != 'passed' or result['build_too_old']:
if result['build_date'] is not None:
email_body += result['platform'] + ': ' +\
result['build_link'] + ' ( Build too old: ' +\
result['build_date'] + ' ) ' +'\n\n'
else:
email_body += result['platform'] + ': ' +\
result['build_link'] + '\n\n'
SendEmailViaSendmailCommand(sender_email, recipient_emails,
subject, email_body)
def SendExceptionEmailViaSendmailCommand(exception_message_lines,
recipient_emails,
sender_email):
subject = EXCEPTION_SUBJECT
email_body = ''
email_body = '\n'.join(exception_message_lines)
SendEmailViaSendmailCommand(sender_email, recipient_emails,
subject, email_body)
class OfficialBuilderParser(object):
"""This class implements basic utility functions on a specified builder."""
def __init__(self, builder_type, build_info):
self.platform = builder_type
self.builder_info = build_info
self.builder_url = build_info['builder_url']
self.build_json_url = build_info['json_url']
self.build = self._GetLatestBuildNumber()
def _GetLatestBuildNumber(self):
json_url = self.builder_info['builds_url']
data = GetJSONData(json_url)
# Get a sorted list of all the keys in the json data.
keys = sorted(map(int, data.keys()))
return self._GetLatestCompletedBuild(keys)
def _GetLatestCompletedBuild(self, keys):
reversed_list = keys[::-1]
for build in reversed_list:
data = self._GetJSONDataForBuild(build)
if data is not None:
if 'text' in data:
return build
return None
def _GetJSONDataForBuild(self, build):
if build is None:
return build
json_url = self.build_json_url % build
return GetJSONData(json_url)
class GetBuilderStatus(OfficialBuilderParser):
def __init__(self, builder_type, build_info):
OfficialBuilderParser.__init__(self, builder_type, build_info)
def CheckForFailedSteps(self, days):
if self.build is None:
return {}
result = {'platform': self.platform,
'build_number': self.build,
'build_link': self.builder_url + str(self.build),
'build_date': None,
'build_too_old': False,
'error': 'unknown'}
data = self._GetJSONDataForBuild(self.build)
if data is not None:
if 'text' in data:
if 'build' in data['text'] and 'successful' in data['text']:
result['error'] = 'passed'
else:
if 'failed' in data['text'] or\
'exception' in data['text'] or\
'interrupted' in data['text']:
result['error'] = 'failed'
if 'times' in data:
old_date = GetTimeDelta(datetime.datetime.now(), days)
last_build_date = GetDateFromEpochFormat(data['times'][0])
if last_build_date < old_date:
result['build_too_old'] = True
result['build_date'] = str(last_build_date).split(' ')[0]
else:
raise Exception('There was some problem getting JSON data '
'from URL: %s' % result['build_link'])
return result
def main():
parser = optparse.OptionParser()
parser.add_option('--config', type='str',
help='Absolute path to the config file.')
(options, _) = parser.parse_args()
if not options.config:
print 'Error: missing required parameter: --config'
parser.print_help()
return 1
try:
with open(options.config, 'r') as config_file:
try:
json_data = json.loads(config_file.read())
except ValueError:
print 'ValueError for loading JSON data from : %s' % options.config
raise ValueError
old_build_days = -2
if 'old_build_days' in json_data:
old_build_days = - json_data['old_build_days']
consolidated_results = []
for key in json_data['build_info'].keys():
builder_status = GetBuilderStatus(key, json_data['build_info'][key])
builder_result = builder_status.CheckForFailedSteps(old_build_days)
consolidated_results.append(builder_result)
SendStatusEmailViaSendmailCommand(consolidated_results,
json_data['recipient_emails'],
json_data['sender_email'])
return 0
except Exception:
formatted_lines = traceback.format_exc().splitlines()
SendExceptionEmailViaSendmailCommand(formatted_lines,
json_data['recipient_emails'],
json_data['sender_email'])
return 1
if __name__ == '__main__':
sys.exit(main())
Name: Google code support upload script
Short Name: googlecode_upload
URL: http://support.googlecode.com/svn/trunk/scripts/googlecode_upload.py
Version: r681
Security Critical: no
License: Apache 2
License File: NOT_SHIPPED
Description:
A simple script from Google Code support for automating uploads.
Local modifications:
-Added comment describing license. They have a comment about the license, but
it uses an atypical format that isn't recognized by chromium's checklicense
script.
-Instead of using the default netrc lookup which requires HOME to be set
(which normally isn't on Windows), specify the file using os.path.expanduser.
-Fail instead of prompting for username/password if none is given.
#!/usr/bin/env python
#
# Copyright 2006, 2007 Google Inc. All Rights Reserved.
# Author: danderson@google.com (David Anderson)
#
# Script for uploading files to a Google Code project.
#
# This is intended to be both a useful script for people who want to
# streamline project uploads and a reference implementation for
# uploading files to Google Code projects.
#
# To upload a file to Google Code, you need to provide a path to the
# file on your local machine, a small summary of what the file is, a
# project name, and a valid account that is a member or owner of that
# project. You can optionally provide a list of labels that apply to
# the file. The file will be uploaded under the same name that it has
# in your local filesystem (that is, the "basename" or last path
# component). Run the script with '--help' to get the exact syntax
# and available options.
#
# Note that the upload script requests that you enter your
# googlecode.com password. This is NOT your Gmail account password!
# This is the password you use on googlecode.com for committing to
# Subversion and uploading files. You can find your password by going
# to http://code.google.com/hosting/settings when logged in with your
# Gmail account. If you have already committed to your project's
# Subversion repository, the script will automatically retrieve your
# credentials from there (unless disabled, see the output of '--help'
# for details).
#
# If you are looking at this script as a reference for implementing
# your own Google Code file uploader, then you should take a look at
# the upload() function, which is the meat of the uploader. You
# basically need to build a multipart/form-data POST request with the
# right fields and send it to https://PROJECT.googlecode.com/files .
# Authenticate the request using HTTP Basic authentication, as is
# shown below.
#
# Licensed under the terms of the Apache Software License 2.0:
# http://www.apache.org/licenses/LICENSE-2.0
#
# Questions, comments, feature requests and patches are most welcome.
# Please direct all of these to the Google Code users group:
# http://groups.google.com/group/google-code-hosting
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
"""Google Code file uploader script.
"""
__author__ = 'danderson@google.com (David Anderson)'
import httplib
import os.path
import optparse
import getpass
import base64
import sys
def upload(file, project_name, user_name, password, summary, labels=None):
"""Upload a file to a Google Code project's file server.
Args:
file: The local path to the file.
project_name: The name of your project on Google Code.
user_name: Your Google account name.
password: The googlecode.com password for your account.
Note that this is NOT your global Google Account password!
summary: A small description for the file.
labels: an optional list of label strings with which to tag the file.
Returns: a tuple:
http_status: 201 if the upload succeeded, something else if an
error occured.
http_reason: The human-readable string associated with http_status
file_url: If the upload succeeded, the URL of the file on Google
Code, None otherwise.
"""
# The login is the user part of user@gmail.com. If the login provided
# is in the full user@domain form, strip it down.
if user_name.endswith('@gmail.com'):
user_name = user_name[:user_name.index('@gmail.com')]
form_fields = [('summary', summary)]
if labels is not None:
form_fields.extend([('label', l.strip()) for l in labels])
content_type, body = encode_upload_request(form_fields, file)
upload_host = '%s.googlecode.com' % project_name
upload_uri = '/files'
auth_token = base64.b64encode('%s:%s'% (user_name, password))
headers = {
'Authorization': 'Basic %s' % auth_token,
'User-Agent': 'Googlecode.com uploader v0.9.4',
'Content-Type': content_type,
}
server = httplib.HTTPSConnection(upload_host)
server.request('POST', upload_uri, body, headers)
resp = server.getresponse()
server.close()
if resp.status == 201:
location = resp.getheader('Location', None)
else:
location = None
return resp.status, resp.reason, location
def encode_upload_request(fields, file_path):
"""Encode the given fields and file into a multipart form body.
fields is a sequence of (name, value) pairs. file is the path of
the file to upload. The file will be uploaded to Google Code with
the same file name.
Returns: (content_type, body) ready for httplib.HTTP instance
"""
BOUNDARY = '----------Googlecode_boundary_reindeer_flotilla'
CRLF = '\r\n'
body = []
# Add the metadata about the upload first
for key, value in fields:
body.extend(
['--' + BOUNDARY,
'Content-Disposition: form-data; name="%s"' % key,
'',
value,
])
# Now add the file itself
file_name = os.path.basename(file_path)
f = open(file_path, 'rb')
file_content = f.read()
f.close()
body.extend(
['--' + BOUNDARY,
'Content-Disposition: form-data; name="filename"; filename="%s"'
% file_name,
# The upload server determines the mime-type, no need to set it.
'Content-Type: application/octet-stream',
'',
file_content,
])
# Finalize the form body
body.extend(['--' + BOUNDARY + '--', ''])
return 'multipart/form-data; boundary=%s' % BOUNDARY, CRLF.join(body)
def upload_find_auth(file_path, project_name, summary, labels=None,
user_name=None, password=None, tries=3):
"""Find credentials and upload a file to a Google Code project's file server.
file_path, project_name, summary, and labels are passed as-is to upload.
Args:
file_path: The local path to the file.
project_name: The name of your project on Google Code.
summary: A small description for the file.
labels: an optional list of label strings with which to tag the file.
config_dir: Path to Subversion configuration directory, 'none', or None.
user_name: Your Google account name.
tries: How many attempts to make.
"""
if user_name is None or password is None:
from netrc import netrc
# Chromium edit: Works on windows without requiring HOME to be set.
netrc_path = os.path.join(os.path.expanduser('~'), '.netrc')
authenticators = netrc(netrc_path).authenticators("code.google.com")
if authenticators:
if user_name is None:
user_name = authenticators[0]
if password is None:
password = authenticators[2]
if user_name is None or password is None:
raise RuntimeError('Missing user credentials for upload')
return upload(file_path, project_name, user_name, password, summary, labels)
def main():
parser = optparse.OptionParser(usage='googlecode-upload.py -s SUMMARY '
'-p PROJECT [options] FILE')
parser.add_option('-s', '--summary', dest='summary',
help='Short description of the file')
parser.add_option('-p', '--project', dest='project',
help='Google Code project name')
parser.add_option('-u', '--user', dest='user',
help='Your Google Code username')
parser.add_option('-w', '--password', dest='password',
help='Your Google Code password')
parser.add_option('-l', '--labels', dest='labels',
help='An optional list of comma-separated labels to attach '
'to the file')
options, args = parser.parse_args()
if not options.summary:
parser.error('File summary is missing.')
elif not options.project:
parser.error('Project name is missing.')
elif len(args) < 1:
parser.error('File to upload not provided.')
elif len(args) > 1:
parser.error('Only one file may be specified.')
file_path = args[0]
if options.labels:
labels = options.labels.split(',')
else:
labels = None
status, reason, url = upload_find_auth(file_path, options.project,
options.summary, labels,
options.user, options.password)
if url:
print 'The file was uploaded successfully.'
print 'URL: %s' % url
return 0
else:
print 'An error occurred. Your file was not uploaded.'
print 'Google Code upload server said: %s (%s)' % (reason, status)
return 1
if __name__ == '__main__':
sys.exit(main())
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment