aws's People
aws's Issues
s3cmd
s3cmd configure
sudo apt-get install s3cmd
s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3
Access Key:
Secret Key:
Save settings? [y/N] y
Configuration saved to '/home/vagrant/.s3cfg'
Put a file
s3cmd put test s3://bucket
test -> s3://bucket/test [1 of 1]
0 of 0 0% in 0s 0.00 B/s done
List a bucket
vagrant@stormbase:/tmp$ s3cmd ls s3://bucket
2013-12-12 18:27 0 s3://bucket/test
RE:Invent 2014
SEC 315 AWS Config
Knowing something changed
polling describe
inventory of resources
vpc, sg, ec2 -> record change -> config item in state version
history
stream -> sns topic
snapshot point in time
audit at point in time
change impact
What changed
Discovery, record config
Bucket
SNS topic
AWS resource is something you can create, update, delete
Relationships
Config Item:
all attributes at a given time
all in json -> jq that
stream sns topic
dash board
alert when something changes in prod
monitor of activity
Time line view
get history of resource in time range
NOT FREE
APP 306 Cloud Formation
JIRA power Deployment
40,000 change tickets
Long feed back loops
high stress - low trust in deployments
large delta between releases
balance of freedom / responsibility
one product team
one playform team -> who is the platform team?
give access make them responsible
Picture of a pendulum one side has predictability and slow, the other has freedom and fast.
repeatability
flexibility
stack overflow ability
confidence
support
access
patterns
reliable
reproducible
Infrastructure versions makes it accessible on net
service.json-0.1.0
ASG
DNS
SG
Roles
ELB
resources.json-0.1.0
DB
S3
SNS
Primitive PO's
Diff accounts
Teams
Services
VPC in CF
full ssh access
bastion in deticated VPC
peered with VPCs
from laptop to live in 2 mins
github/bbs
BBC has a Project named wormhole
Platform team delivers interface to devs
The opposite of what we have
devs deliver platform
software snapshot
apply config snapshot
tag amis. one ami with software, one with config
tropshere
ansible cloud formation
use ansible to create cloud formation python
bakery
1 snapshot with software bits
1 snapshot with configs
ASG
Update Policy
Test
Live
Indented state in version control
push to S3 for HA
consume with IAM role
customer template
Business driven development
ARC 307 Infrastructure as code
Consistency
SG disaster porn
SG define in group_vars for each stack
DEV | Infra | OPS
Grace Hopper quote, most dangerous phrase is "we've always done it like that"
What infra should be model of AWS resources
Reality reflects model. Audit for diffs apply changes.
BDT 403
10 PB data warehouse in S3
event to s3 15 mins
Log -> suro -> ursula -> s3
Log -> cassandra -> sstables -> aegiela -> s3
Storage seperate from compute
Parquet file format in S3
index data to s3
m1s - netflix uses them all
Presto interactive SQL query from s3 data
techblog.netflix.com
cache work
"needle in hay stacl"
query graph
5% - 10% speed diff S3 vs HDFS
not tried ssds
open source job inspector Inviso
BDT 310
Volume Up
Velocity Up
Variety Up
Real Time Big Data
Data -> ingest -> Store -> Process -> Visualize Answers
Over Time ->
S3 Filesystem + indexer try it out
Cheat Slide. A big list of vendors and open source tools for each component.
Ingest
Store
Process/ Analyze
Visualize
Where is splunk
Types of Data Ingest
Transacrtional
File
Stream
Store
Amazon Kinesis
Apache Kafika
Convert multiple streams into few
Decouple producers / consumers
ordering preserved
replay
What is splunks interface to hot, cold, warm?
How can I inject s3?
abstract buckets to s3
Data characteristics
/hot - ssd, cache, ebs
/warm - s3, file system
/cold - glacier file system
try this too
amazone elastic cache
dynamo DB
hdfs for hot
segment the volumes
Processing
Batch
Stream
Processing tools
Presto
open source
vendors
Visualize
Open source
vendors
Where is splunk?
Write to multiple data stores
Data -> indexer -> DB & S3
get splunk on ingest and store
glacier_cold
s3_warm
HDFS_hot
Ganglia for stacks
MBL 202 Containers
APP 313 Lambda
AWS
- Region
- VPC
- Availability Zone
- Multiple Subnets VLAN Zones
- App
- DB
- LB
- Security Groups
- Allow rules only
- Network Access Control List (NACL)
Use jump hosts to connect.
IAM
IAM
IAM roles for access, boto loads them automatically.
>>> import boto
>>> c = boto.connect_s3()
>>> rs = c.get_all_buckets()
>>> print rs
[<Bucket: cf-templates-9972b0b8kz5j-us-east-1>, <Bucket: elasticbeanstalk-us-east-1-811769099702>]
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.