Quora技术构架

Feb 09 2011

现在国内正在上限一个站点,叫知乎,直接是抄袭的Quora。这里Phil Whelan给出了关于Quora技术构架的详解,在一些地方看到,现在的大网站用户量、数据量上去以后,构架就那几个选择,出创新的地方不多,的确,也许解析中的一些东西不新,但是对于我们这些希望以后能够独挡一面的童鞋来说,算是进步台阶上的一个垫脚石吧。

对于Quora,Phil首先说到的就是速度,然后延伸为搜索框、long polling、缓存,然后就是说到了关于服务器软件,云端,编程语言等等

      • 搜索框方面,Quora因为只是为标签、用户名、标题建立了索引,所以,你看到的快速反应时自然地,因为全文检索现在没加入进去,而如果现在加入,估计响应速度就要降几个数量级了。同时,全文检索没有,只是依靠标题来进行相关性的提示,不算是完善,以后改进的重点估计会在这里。
  • long polling,因为我们是Web 2.0的世道,所以什么刷新呀越少越好,自然地、无缝的交互不断被提倡,方式就是使用Ajax来更新,但是,这里就有问题了,局部更新你需要不断http请求,少了还可以,但是多了呢?Phil举个例子,在他们将近1百万的用户中,如果有10万人在线呢?你的服务器需要接受到10万个请求,并且如果一个页面中请求的连接多、请求频繁,服务器负载就不简单控制了。所以,Phil说“使用long polling呀”,恩呢,这就是我一直想要寻求的技术(嘿嘿,我是技术新手,不懂怎么描述它)~设置在50秒内等待请求,极大地减缓了服务器压力,不过你也是最多把它减少10万*n多点,所以,如果你的服务器端软件不行那就嗝屁了
  • 缓存,通过使用缓存能够将动态的网页简单的变成了半静态了,Memcached还是首选,因为后台他们使用的MySql作为数据存储项
  • 负载均衡,使用HAProxy呗
  • 反向代理,Nginx
  • 对了,忘记重点了,他们使用Python作为大部分的构架的主力语言,因为用的多,跨构架多,差点就用ruby,可惜都不熟悉,看来ruby的人气不是一般的旺
  • 而在网页构建方面,他们使用的Amazon S3来保存静态文件,同时遵循了14条快速加载网页的要领:
  • 更少的HTTP请求
    使用内容传输网络
    添加过期头
    Gzip
    头部添加网页风格文件
    底部放置javascript文件
    Avoid CSS Expressions
    外部放置Javascript和CSS
    减少DNS查找(难道是直接10.0.186.1?域名IP换了怎么办?)
    最小化Javascript
    拒绝转向
    删除没用的脚本
    配置ETags
    缓存AJAX

原文地址:http://www.philwhln.com/quoras-technology-examined

One response so far

MVC步步深入:11

Jan 30 2011

关于数据库的ORM方便,前面的几个例子都是使用的Entity Frame,这个东西比NHibrate起步晚,而且还正在发展壮大中,所以使用的成熟度还有待考验,而微软自己的一个示范项目中,貌似就是使用的NHibrate而非EF,但是我们不能否认EF与微软产品结合的程度要高一些,并且近期EF CTP5的发布算是为这个产品吹鼓东风吧

因为很多概念可能没有介绍过,所以下面先给出完整的代码,以后的几篇文章都是根据这个代码来做解释,当然这是根据微软官方的文章做基础,不是原创。

先说下背景,有这样几个类:Princess, Unicorn, Castle, and LadyInWaiting。其中Princess能够拥有很多Unicorn、Castle以及Ladyinwaiting,但是一个Castle中只能有一个Ladyinwaiting。每一个Castle都是按照位置来做标记,他具有一个符合类型ImaginaryWorld。

完整的Model代码:

using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;
using System.Data;
using System.Data.Entity;
using System.Data.Entity.Database;
using System.Data.Entity.Infrastructure;
using System.Data.Metadata.Edm;
using System.Data.Objects;
using System.Linq;

namespace Magic.Unicorn
{
    public class Princess : IPerson
    {
        public int Id { get; set; }
        public string Name { get; set; }

        public virtual ICollection Unicorns { get; set; }
        public virtual ICollection LadiesInWaiting { get; set; }
    }

    public class Unicorn
    {
        public int Id { get; set; }
        public string Name { get; set; }

        [Timestamp]
        public byte[] Version { get; set; }

        public int PrincessId { get; set; } // FK for Princess reference
        public virtual Princess Princess { get; set; }
    }

    public class Castle
    {
        [Key]
        public string Name { get; set; }

        public Location Location { get; set; }

        public virtual ICollection LadiesInWaiting { get; set; }
    }

    [ComplexType]
    public class Location
    {
        public string City { get; set; }
        public string Kingdom { get; set; }

        public ImaginaryWorld ImaginaryWorld { get; set; }
    }

    [ComplexType]
    public class ImaginaryWorld
    {
        public string Name { get; set; }
        public string Creator { get; set; }
    }

    public class LadyInWaiting : IPerson
    {
        [Key, Column(Order = 0)]
        [DatabaseGenerated(DatabaseGenerationOption.None)]
        public int PrincessId { get; set; } // FK for Princess reference

        [Key, Column(Order = 1)]
        public string CastleName { get; set; } // FK for Castle reference

        public string FirstName { get; set; }
        public string Title { get; set; }

        [NotMapped]
        public string Name
        {
            get
            {
                return String.Format("{0} {1}", Title, FirstName);
            }
        }

        public virtual Castle Castle { get; set; }
        public virtual Princess Princess { get; set; }
    }

    public interface IPerson
    {
        string Name { get; }
    }

    public class UnicornsContext : DbContext
    {
        public DbSet Unicorns { get; set; }
        public DbSet
 Princesses { get; set; }
        public DbSet LadiesInWaiting { get; set; }
        public DbSet Castles { get; set; }
    }

    public class UnicornsContextInitializer
        : DropCreateDatabaseAlways
    {
        protected override void Seed(UnicornsContext context)
        {
            var cinderella = new Princess { Name = "Cinderella" };
            var sleepingBeauty = new Princess { Name = "Sleeping Beauty" };
            var snowWhite = new Princess { Name = "Snow White" };

            new List
            {
                new Unicorn { Name = "Binky" , Princess = cinderella },
                new Unicorn { Name = "Silly" , Princess = cinderella },
                new Unicorn { Name = "Beepy" , Princess = sleepingBeauty },
                new Unicorn { Name = "Creepy" , Princess = snowWhite }
            }.ForEach(u => context.Unicorns.Add(u));

            var efCastle = new Castle
            {
                Name = "The EF Castle",
                Location = new Location
                {
                    City = "Redmond",
                    Kingdom = "Rainier",
                    ImaginaryWorld = new ImaginaryWorld
                    {
                        Name = "Magic Unicorn World",
                        Creator = "ADO.NET"
                    }
                },
            };

            new List
            {
                new LadyInWaiting { Princess = cinderella,
                                    Castle = efCastle,
                                    FirstName = "Lettice",
                                    Title = "Countess" },
                new LadyInWaiting { Princess = sleepingBeauty,
                                    Castle = efCastle,
                                    FirstName = "Ulrika",
                                    Title = "Lady" },
                new LadyInWaiting { Princess = snowWhite,
                                    Castle = efCastle,
                                    FirstName = "Yolande",
                                    Title = "Duchess" }
            }.ForEach(l => context.LadiesInWaiting.Add(l));
        }
    }

    public class Program
    {
        public static void Main(string[] args)
        {
            DbDatabase.SetInitializer(new UnicornsContextInitializer());

            // Many of the code fragments can be run by inserting them here
        }
    }
}

One response so far

MVC步步深入:10

Jan 29 2011

当用户购物完成以后就需要进行付款了,但是我们前面说到了,我们允许没有登陆的用户购买,所以在付款的时候需要验证是否已经登陆。

如果登陆了就好,但是没有登陆就需要进行注册了。当用户注册以后,因为开始是匿名购买,所以需要合并订单到现在的用户中来:

private void MigrateShoppingCart(string UserName)
 {
    // Associate shopping cart items with logged-in user
    var cart = ShoppingCart.GetCart(this.HttpContext);

    cart.MigrateCart(UserName);
    Session[ShoppingCart.CartSessionKey] = UserName;
 }

下面是登录函数:

[HttpPost]
 public ActionResult LogOn(LogOnModel model, string returnUrl)
 {
    if (ModelState.IsValid)
    {
        if (MembershipService.ValidateUser(model.UserName, model.Password))
        {
            MigrateShoppingCart(model.UserName);

            FormsService.SignIn(model.UserName, model.RememberMe);
            if (Url.IsLocalUrl(returnUrl))
            {
                return Redirect(returnUrl);
            }
            else
            {
                return RedirectToAction("Index", "Home");
            }
        }
        else
        {
            ModelState.AddModelError("", "The user name or
password provided is incorrect.");
        }
    }
    // If we got this far, something failed, redisplay form
    return View(model);
 }

以及注册函数:

[HttpPost]
 public ActionResult Register(RegisterModel model)
 {
    if (ModelState.IsValid)
    {
        // Attempt to register the user
        MembershipCreateStatus createStatus =
MembershipService.CreateUser(model.UserName, model.Password, model.Email);

        if (createStatus == MembershipCreateStatus.Success)
        {
            MigrateShoppingCart(model.UserName);

            FormsService.SignIn(model.UserName, false /* createPersistentCookie
*/);
            return RedirectToAction("Index", "Home");
        }
        else
        {
            ModelState.AddModelError("", AccountValidation.ErrorCodeToString(createStatus));
        }
    }
    // If we got this far, something failed, redisplay form
    ViewBag.PasswordLength = MembershipService.MinPasswordLength;
    return View(model);
 }

紧接着的付款需要使用付款控制器,首先自然是创建一个啦:

using System;
 using System.Linq;
 using System.Web.Mvc;
 using MvcMusicStore.Models;

namespace MvcMusicStore.Controllers
{
    [Authorize]
    public class CheckoutController : Controller
    {
        MusicStoreEntities storeDB = new MusicStoreEntities();
        const string PromoCode = "FREE";

因为我们其实不是要实现真的付款过程,所以使用一个促销码来代替。

//
// GET: /Checkout/AddressAndPayment
public ActionResult AddressAndPayment()
{
    return View();
}
//
// POST: /Checkout/AddressAndPayment
[HttpPost]
 public ActionResult AddressAndPayment(FormCollection values)
 {
    var order = new Order();
    TryUpdateModel(order);

    try
    {
        if (string.Equals(values["PromoCode"], PromoCode,
            StringComparison.OrdinalIgnoreCase) == false)
        {
            return View(order);
        }
        else
        {
            order.Username = User.Identity.Name;
            order.OrderDate = DateTime.Now;

            //Save Order
            storeDB.AddToOrders(order);
            storeDB.SaveChanges();
            //Process the order
            var cart = ShoppingCart.GetCart(this.HttpContext);
            cart.CreateOrder(order);

            return RedirectToAction("Complete",
                new { id = order.OrderId });
        }
    }
    catch
    {
        //Invalid - redisplay with errors
        return View(order);
    }
}

当完成以后,我们就给个提示:

//
// GET: /Checkout/Complete
public ActionResult Complete(int id)
 {
    // Validate customer owns this order
    bool isValid = storeDB.Orders.Any(
        o => o.OrderId == id &&
        o.Username == User.Identity.Name);

    if (isValid)
    {
        return View(id);
    }
    else
    {
        return View("Error");
    }
}

最后就是添加视图,这个很简单,大家都会,不过因为需要验证,所以,我们复习下验证的特性:

using System.Collections.Generic;
 using System.ComponentModel;
 using System.ComponentModel.DataAnnotations;
 using System.Web.Mvc;

namespace MvcMusicStore.Models
{
    [Bind(Exclude = "OrderId")]
    public partial class Order
    {
        [ScaffoldColumn(false)]
        public int OrderId { get; set; }
        [ScaffoldColumn(false)]
        public System.DateTime OrderDate { get; set; }
        [ScaffoldColumn(false)]
        public string Username { get; set; }
        [Required(ErrorMessage = "First Name is required")]
        [DisplayName("First Name")]
        [StringLength(160)]
        public string FirstName { get; set; }
        [Required(ErrorMessage = "Last Name is required")]
        [DisplayName("Last Name")]
        [StringLength(160)]
        public string LastName { get; set; }
        [Required(ErrorMessage = "Address is required")]
        [StringLength(70)]
        public string Address { get; set; }
        [Required(ErrorMessage = "City is required")]
        [StringLength(40)]
        public string City { get; set; }
        [Required(ErrorMessage = "State is required")]
        [StringLength(40)]
        public string State { get; set; }
        [Required(ErrorMessage = "Postal Code is required")]
        [DisplayName("Postal Code")]
        [StringLength(10)]
        public string PostalCode { get; set; }
        [Required(ErrorMessage = "Country is required")]
        [StringLength(40)]
        public string Country { get; set; }
        [Required(ErrorMessage = "Phone is required")]
        [StringLength(24)]
        public string Phone { get; set; }
        [Required(ErrorMessage = "Email Address is required")]
        [DisplayName("Email Address")]
        [RegularExpression(@"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}",
            ErrorMessage = "Email is is not valid.")]
        [DataType(DataType.EmailAddress)]
        public string Email { get; set; }
        [ScaffoldColumn(false)]
        public decimal Total { get; set; }
        public List OrderDetails { get; set; }
    }
}

最后就是更新一下错误提示页面:

@{
    ViewBag.Title = "Error";
}

Error

We're sorry, we've hit an unexpected error. Click here if you'd like to go back and try that again.

现在我们的音乐商店基本上已经完工了,最后还一步骤就是添加导航。

接下来,添加信息提示的局部页面:

12

//
// GET: /ShoppingCart/CartSummary
[ChildActionOnly]
 public ActionResult CartSummary()
{
    var cart = ShoppingCart.GetCart(this.HttpContext);

    ViewData["CartCount"] = cart.GetCount();
    return PartialView("CartSummary");
}

注意到这个ChildActionOnly没?他是必须的,这样页面才知道这一款相当于一个局部的模板。

修改前台:

@Html.ActionLink("Cart
(" + ViewData["CartCount"] + ")",
    "Index",
    "ShoppingCart",
    new { id = "cart-status" })

同时我们还可以通过

@Html.RenderAction("CartSummary", "ShoppingCart")在页面的其他地方来显示这个小局部页面。
接下来再创建菜单栏,如图:
1
/
// GET: /Store/GenreMenu
[ChildActionOnly]
 public ActionResult GenreMenu()
{
    var genres = storeDB.Genres.ToList();
    return PartialView(genres);
 }
然后更新总的模板:



    


    @{Html.RenderAction("GenreMenu",
"Store");}
@RenderBody()
更新浏览页面的模板:
@model MvcMusicStore.Models.Genre

@{
    ViewBag.Title = "Browse Albums";
}

@Model.Name Albums

21

下面我们再提出一个小小的要求,那就是置顶畅销的专辑。

更新一下Album类:

public virtual Genre  Genre                     { get; set;
}
        public virtual Artist Artist                    { get; set; }
        public virtual List
OrderDetails   { get; set; }
    }
}

然后对于HomeController做出如下的修改:

using System.Collections.Generic;
 using System.Linq;
 using System.Web.Mvc;
 using MvcMusicStore.Models;

namespace MvcMusicStore.Controllers
{
    public class HomeController : Controller
    {
        //
        // GET: /Home/
        MusicStoreEntities storeDB = new MusicStoreEntities();
        public ActionResult Index()
        {
            // Get most popular albums
            var albums = GetTopSellingAlbums(5);

            return View(albums);
        }
        private List GetTopSellingAlbums(int count)
        {
            // Group the order details by album and return
            // the albums with the highest count
            return storeDB.Albums
                .OrderByDescending(a => a.OrderDetails.Count())
                .Take(count)
                .ToList();
        }
    }
}

最后就是更新视图:让他能够正确显示这些专辑

@model List
@{
    ViewBag.Title = "ASP.NET MVC Music Store";
}

Fresh off the grill

11

这就是最后的成果,帅吧~呜哈哈,其实MVC的功能比这个更加强大~我们以后会慢慢说来,这算是第二个阶段结束了,新的“冒险”开始了~开船吧~

2 responses so far

« Newer - Older »