StarCluster is an open source cluster-computing toolkit for Amazon's Elastic Compute Cloud (EC2).

jtriley, updated 🕥 2022-03-29 21:53:54

=================== StarCluster v0.95.6 =================== :StarCluster: Cluster Computing Toolkit for the Cloud :Version: 0.95.6 :Author: Justin Riley justin.t.riley@gmail.com :Team: Software Tools for Academics and Researchers (http://star.mit.edu) :Homepage: http://star.mit.edu/cluster :License: LGPL .. image:: https://secure.travis-ci.org/jtriley/StarCluster.png?branch=develop :target: https://secure.travis-ci.org/jtriley/StarCluster .. image:: https://pypip.in/d/StarCluster/badge.png :target: https://crate.io/packages/StarCluster

Description:

StarCluster is a utility for creating and managing computing clusters hosted on Amazon's Elastic Compute Cloud (EC2). StarCluster utilizes Amazon's EC2 web service to create and destroy clusters of Linux virtual machines on demand.

All that's needed to create your own cluster(s) on Amazon EC2 is an AWS account and StarCluster. StarCluster features:

Simple configuration - with examples ready to go out-of-the-box
Create/Manage Clusters - simple start command to automatically launch and configure one or more clusters on EC2
Automated Cluster Setup - includes NFS-sharing, Open Grid Scheduler queuing system, Condor, password-less ssh between machines, and more
Scientific Computing AMI - comes with Ubuntu 11.10-based EBS-backed AMI that contains Hadoop, OpenMPI, ATLAS, LAPACK, NumPy, SciPy, IPython, and other useful libraries
EBS Volume Sharing - easily NFS-share Amazon Elastic Block Storage (EBS) volumes across a cluster for persistent storage
EBS-Backed Clusters - start and stop EBS-backed clusters on EC2
Cluster Compute Instances - support for "cluster compute" instance types
Expand/Shrink Clusters - scale a cluster by adding or removing nodes
Elastic Load Balancing - automatically shrink or expand a cluster based on Open Grid Scheduler queue statistics
Plugin Support - allows users to run additional setup routines on the cluster after StarCluster's defaults. Comes with plugins for IPython parallel+notebook, Condor, Hadoop, MPICH2, MySQL cluster, installing Ubuntu packages, and more.

Interested? See the getting started_ section for more details.

.. _getting started:

Getting Started:

Install StarCluster using easy_install::

$ easy_install StarCluster

or using pip::

$ pip install StarCluster

or manually::

$ (Download StarCluster from http://star.mit.edu/cluster/downloads.html)
$ tar xvzf starcluster-X.X.X.tar.gz  (where x.x.x is a version number)
$ cd starcluster-X.X.X
$ sudo python setup.py install

After the software has been installed, the next step is to setup the configuration file::

$ starcluster help
StarCluster - (http://star.mit.edu/cluster)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to [email protected]

!!! ERROR - config file /home/user/.starcluster/config does not exist

Options:
--------
[1] Show the StarCluster config template
[2] Write config template to /home/user/.starcluster/config
[q] Quit

Please enter your selection:

Select the second option by typing 2 and pressing enter. This will give you a template to use to create a configuration file containing your AWS credentials, cluster settings, etc. The next step is to customize this file using your favorite text-editor::

$ vi ~/.starcluster/config

This file is commented with example "cluster templates". A cluster template defines a set of configuration settings used to start a new cluster. The example config provides a smallcluster template that is ready to go out-of-the-box. However, first, you must fill in your AWS credentials and keypair info::

[aws info]
aws_access_key_id = #your aws access key id here
aws_secret_access_key = #your secret aws access key here
aws_user_id = #your 12-digit aws user id here

The next step is to fill in your keypair information. If you don't already have a keypair you can create one from StarCluster using::

$ starcluster createkey mykey -o ~/.ssh/mykey.rsa

This will create a keypair called mykey on Amazon EC2 and save the private key to ~/.ssh/mykey.rsa. Once you have a key the next step is to fill-in your keypair info in the StarCluster config file::

[key key-name-here]
key_location = /path/to/your/keypair.rsa

For example, the section for the keypair created above using the createkey command would look like::

[key mykey]
key_location = ~/.ssh/mykey.rsa

After defining your keypair in the config, the next step is to update the default cluster template smallcluster with the name of your keypair on EC2::

[cluster smallcluster]
keyname = key-name-here

For example, the smallcluster template would be updated to look like::

[cluster smallcluster]
keyname = mykey

Now that the config file has been set up we're ready to start using StarCluster. Next we start a cluster named "mycluster" using the default cluster template smallcluster in the example config::

$ starcluster start mycluster

The default_template setting in the [global] section of the config specifies the default cluster template and is automatically set to smallcluster in the example config.

After the start command completes you should now have a working cluster. You can login to the master node as root by running::

$ starcluster sshmaster mycluster

You can also copy files to/from the cluster using the put and get commands. To copy a file or entire directory from your local computer to the cluster::

$ starcluster put mycluster /path/to/local/file/or/dir /remote/path/

To copy a file or an entire directory from the cluster to your local computer::

$ starcluster get mycluster /path/to/remote/file/or/dir /local/path/

Once you've finished using the cluster and wish to stop paying for it::

$ starcluster terminate mycluster

Have a look at the rest of StarCluster's available commands::

$ starcluster --help

Dependencies:

Amazon AWS Account
Python 2.6+
Boto 2.23.0+
Paramiko 1.12.1+
WorkerPool 0.9.2
Jinja2 2.7
decorator 3.4.0+
iptools 0.6.1+
optcomplete 1.2-devel+
PyCrypto 2.5+
scp 0.7.1+
iso8601 0.1.8+

Learn more...

Watch an ~8 minute screencast @ http://star.mit.edu/cluster

To learn more have a look at the documentation: http://star.mit.edu/cluster/docs/latest

Community

StarCluster has a mailing list for users and developers:

http://star.mit.edu/cluster/mailinglist.html

Join our IRC channel #starcluster on freenode. If you do not have an IRC client you can join the #starcluster channel using your web browser:

http://webchat.freenode.net/?channels=starcluster

Licensing

StarCluster is licensed under the LGPLv3 See COPYING.LESSER (LGPL) and COPYING (GPL) for LICENSE details

Issues

build(deps): bump paramiko from 1.12.1 to 2.10.1

opened on 2022-03-29 21:53:54 by dependabot[bot]

Bumps paramiko from 1.12.1 to 2.10.1.

Commits

286bd9f Cut 2.10.1
4c491e2 Fix CVE re: PKey.write_private_key chmod race
aa3cc6f Cut 2.10.0
e50e19f Fix up changelog entry with real links
02ad67e Helps to actually leverage your mocked system calls
29d7bf4 Clearly our agent stuff is not fully tested yet...
5fcb8da OpenSSH docs state %C should also work in IdentityFile and Match exec
1bf3dce Changelog enhancement
f6342fc Prettify, add %C as acceptable controlpath token, mock gethostname
3f3451f Add to changelog
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/jtriley/StarCluster/network/alerts).

build(deps): bump ipython from 1.1.0 to 7.16.3

opened on 2022-01-21 18:56:23 by dependabot[bot]

Bumps ipython from 1.1.0 to 7.16.3.

Release notes

Sourced from ipython's releases.

7.9.0

No release notes provided.

7.8.0

No release notes provided.

7.7.0

No release notes provided.

7.6.1

No release notes provided.

7.6.0

No release notes provided.

7.5.0

No release notes provided.

7.4.0

No release notes provided.

7.3.0

No release notes provided.

7.2.0

No release notes provided.

7.1.1

No release notes provided.

7.1.0

No release notes provided.

7.0.1

No release notes provided.

7.0.0

No release notes provided.

7.0.0-doc

No release notes provided.

7.0.0rc1

No release notes provided.

7.0.0b1

No release notes provided.

6.2.1

No release notes provided.

... (truncated)

Commits

d43c7c7 release 7.16.3
5fa1e40 Merge pull request from GHSA-pq7m-3gw7-gq5x
8df8971 back to dev
9f477b7 release 7.16.2
138f266 bring back release helper from master branch
5aa3634 Merge pull request #13341 from meeseeksmachine/auto-backport-of-pr-13335-on-7...
bcae8e0 Backport PR #13335: What's new 7.16.2
8fcdcd3 Pin Jedi to <0.17.2.
2486838 release 7.16.1
20bdc6f fix conda build
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Create qasem

opened on 2021-05-20 20:01:48 by q-o-ap None

build(deps): bump jinja2 from 2.7.2 to 2.11.3

opened on 2021-03-19 21:36:51 by dependabot[bot]

Bumps jinja2 from 2.7.2 to 2.11.3.

Release notes

Sourced from jinja2's releases.

2.11.3

This contains a fix for a speed issue with the urlize filter. urlize is likely to be called on untrusted user input. For certain inputs some of the regular expressions used to parse the text could take a very long time due to backtracking. As part of the fix, the email matching became slightly stricter. The various speedups apply to urlize in general, not just the specific input cases.

PyPI: https://pypi.org/project/Jinja2/2.11.3/

Changes: https://jinja.palletsprojects.com/en/2.11.x/changelog/#version-2-11-3

2.11.2

Changelog: https://jinja.palletsprojects.com/en/2.11.x/changelog/#version-2-11-2

2.11.1

This fixes an issue in async environment when indexing the result of an attribute lookup, like {{ data.items[1:] }}.

Changes: https://jinja.palletsprojects.com/en/2.11.x/changelog/#version-2-11-1

2.11.0

Changes: https://jinja.palletsprojects.com/en/2.11.x/changelog/#version-2-11-0

Blog: https://palletsprojects.com/blog/jinja-2-11-0-released/

Twitter: https://twitter.com/PalletsTeam/status/1221883554537230336

This is the last version to support Python 2.7 and 3.5. The next version will be Jinja 3.0 and will support Python 3.6 and newer.

2.10.3

Changes: http://jinja.palletsprojects.com/en/2.10.x/changelog/#version-2-10-3

2.10.2

Changes: http://jinja.palletsprojects.com/en/2.10.x/changelog/#version-2-10-2

2.10.1

Changes: https://jinja.palletsprojects.com/en/2.10.x/changelog/#version-2-10-1

Blog: https://palletsprojects.com/blog/jinja-2-10-1-released/

Twitter: https://twitter.com/PalletsTeam/status/1114605127308992513

2.10

Primary changes

A NativeEnvironment that renders Python types instead of strings. http://jinja.pocoo.org/docs/2.10/nativetypes/

A namespace object that works with {% set %}. This replaces previous hacks for storing state across iterations or scopes. http://jinja.pocoo.org/docs/2.10/templates/#assignments

The loop object now has nextitem and previtem attributes, as well as a changed method, for the common case of outputting something as a value in the loop changes. More complicated cases can use the namespace object. http://jinja.pocoo.org/docs/2.10/templates/#for

Install or upgrade

Install from PyPI with pip:

... (truncated)

Changelog

Sourced from jinja2's changelog.

Version 2.11.3

Released 2021-01-31

Improve the speed of the urlize filter by reducing regex backtracking. Email matching requires a word character at the start of the domain part, and only word characters in the TLD. :pr:1343

Version 2.11.2

Released 2020-04-13

Fix a bug that caused callable objects with __getattr__, like :class:~unittest.mock.Mock to be treated as a :func:contextfunction. :issue:1145

Update wordcount filter to trigger :class:Undefined methods by wrapping the input in :func:soft_str. :pr:1160

Fix a hang when displaying tracebacks on Python 32-bit. :issue:1162

Showing an undefined error for an object that raises AttributeError on access doesn't cause a recursion error. :issue:1177

Revert changes to :class:~loaders.PackageLoader from 2.10 which removed the dependency on setuptools and pkg_resources, and added limited support for namespace packages. The changes caused issues when using Pytest. Due to the difficulty in supporting Python 2 and :pep:451 simultaneously, the changes are reverted until 3.0. :pr:1182

Fix line numbers in error messages when newlines are stripped. :pr:1178

The special namespace() assignment object in templates works in async environments. :issue:1180

Fix whitespace being removed before tags in the middle of lines when lstrip_blocks is enabled. :issue:1138

:class:~nativetypes.NativeEnvironment doesn't evaluate intermediate strings during rendering. This prevents early evaluation which could change the value of an expression. :issue:1186

Version 2.11.1

Released 2020-01-30

Fix a bug that prevented looking up a key after an attribute ({{ data.items[1:] }}) in an async template. :issue:1141

... (truncated)

Commits

cf21539 release version 2.11.3
15ef8f0 Merge pull request #1343 from pallets/urlize-speedup
ef658dc speed up urlize matching
eeca0fe Merge pull request #1207 from mhansen/patch-1
2dd7691 Merge pull request #1209 from mhansen/patch-3
4892940 do_dictsort: update example ready to copy/paste
7db7d33 api.rst: bugfix in docs, import PackageLoader
9ec465b fix changelog header
737a4cd release version 2.11.2
179df6b Merge pull request #1190 from pallets/native-eval
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Issue installing on macOS Mojave v10.14.5

opened on 2019-05-28 01:52:25 by oscaco

Hi all, I've been getting this error when trying to install via easy_install.

cc -bundle -undefined dynamic_lookup -Wl,-F. build/temp.macosx-10.14-intel-2.7/build/temp.macosx-10.14-intel-2.7/_openssl.o -lssl -lcrypto -o build/lib.macosx-10.14-intel-2.7/cryptography/hazmat/bindings/_openssl.so ld: library not found for -lssl clang: error: linker command failed with exit code 1 (use -v to see invocation) error: Setup script exited with error: command 'cc' failed with exit status 1

Anyone encounter something like this, any and all help would be appreciated :).

renaming master and node names

opened on 2019-01-30 07:20:45 by tamuanand

Is it possible to have a command line argument that can prefix the master node names and worker node names to something different?

For example, lets say I am using StarCluster to start a cluster of 3, right now it has the names as master, node001, node002. Is it possible here to have the names as prefix_master, prefix_node001, prefix_node002

Thanks in advance, Anand

Justin Riley

GitHub Repository Homepage